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ABSTRACT 

Least squares and Hayes methcds tseie used in a cross 
validation study conducted for comparison parpo;^es. The study applies 
to situations with the following conditicEs: predictor data are given 
on the same scales; criterion data may c€ given on different scales; 
and it is necessary to pool data even though criterion scale 
differences exist. Such a system may te needed for ffinority group 
prediction studies or qraduate school pi^edicticn studies %here the 
group sizes are small. Data for the study were taken frcB the files 
of the Validity Study Service of the Ccllege Entrance Examination 
Board. Also a very limited amount of data .were supplied ty a few 
American qraduate schools. The Bayes cethod was somewhat tetter^ but 
it was found that both methods yielded negative regression weights; 
when the absolute values of the weights were used, tcth tfce methods 
were improved and yielded results which were very similar in terms of 
evaluative statistics computed in the cross validation sample, 
(Author/CTM) 
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COMPARISON OF A BAYFSIAN' AND A LEAST SQUARES 
METHOD OF KDUCATION/VL PRKDICTTON 

A bs t race 

The prediction systems under discussion apply where the following 
conditions obtain: Predictor data are given on the same scale, criterion 
scores may be given on different scales, and it is necessary to pool data 
even though criterion scale differences exist. Such a system may be 
needed for minority group or graduate student prediction where the group 
sizes are small. Least squares and Bayes methods are used in a cross- 
validation study conducted for comparison purposes. Data for the study 
were taken from the files of the Validity Study Service of the College 
Entrance Examination Board. A very limited amount of data were supplied 
by a few American graduate schools. The Bayes method was better, but it 
was found that both riethods yield negative regression weights; when the 
absolute values of the weights were used, the methods were both improved 
and yielded results which were very similar in terms of evaluative sta- 
tistics computed in the cross sample. 



C:OMPARLSr»N OF A BAYESIAN AND A LEAST SQUARES 
METHOD OF EDUCATIONAL PREDICTION"^ 

LnLrod i ic f Ion 

Very often the evaluation of the effectiveness of test scores for pre- 
diction is infeasible because those interested in such evaluation are unable 
to assemble a group of examinees for whom comparable criterion scores are 
available. In some population segments, sucli as minority groups, graduate 
students, and possibly various occupational groups, one often cannot find 
enough people at a single place where an acceptable criterion exists to con- 
duct a statistical study of the predictive validity of selection instruments, 
or at least a stucV in whose results one can have confidence. It is more 
common to find small groups from the population of interest interspersed 
through a variety of locations, performing tasks that seem reasonably 
similar. Evaluation of the performances is made with reference to the group 
at a location but without reference to performances outside of that group. 
Thus, the groups may differ from each other in terms of average performance 
or in the variation in performance, but these differences may not be infera- 
ble trom the corresponding statistics calculated using quantitative evalua- 
tions of performances made at each location. This type of problem was 
encountered in /i study of the use of the Prueba de Aptitud Academica (PAA.) 
in predicting the success of Spanish-speaking students in American universi- 
ties (Gannon, Oppenheim, ^ Wohlhueter, 1966). In that study, efforts to 
accumulate usable data from six locations that apparently promised a reason- 
able supply of Spanish-speaking students yielded Spanish-speaking U.S. 
citizens in group s izes of 72, 8, and 15 , and nonci tizens in group sizes 
of 22, 2.3, 6, 27, and 32. Because of such low numbers of available cases. 



ono would us'.iallv .il)ori a snidv in which only SLM^iarri valid-itv stiiu.iL'S at 
each locit ion were c<)n l onip la I c<.i . In a lat:t.:'r aitt^mpl to repeat tlie study by 
'/Tarip.on a/I . , (Ir. rlso.n ri^^Tw") iia.d lo thjal with •xzr(^rn'^ i ^v.'S as 1 nv a-; n i nc^ . 

Sir.iilrir prni>loms were encoiin tere^.i in the d(*ve 1 opnt;n t c^f the (.'ompa ra t i ve 
(h.iid.\nco and Plaeement (f-CrP) hatterv of tlie ('olI.e,^^e ia'}t ranee llxani na t i on 
Board (CKKIO 5 whicJi is a battery of psycho 1 of^i ca 1 tests intended for [guidance 
-and pLact.^ment use in American junior colIer,es. One researcii problem in the 
devo; 1 opnu.^n t ot this battery w/is tc^ c:hoose a set (^t test.s that v/ould be \'al id 
Tor nr^Hj i t in. y.; success in ea^^di of a niinlw^'r of curricula, Al tliou.r?;}-! anv one 
junior co Lie, v^e would h.a*/e a freshman class large cnoa^2;h to use in conducting 
a study ^ tliat class, wiien broken down by curricula, would becoine liighly 
fractionated. Restrictions in class size were also necessary in order to 
accommodate institutions with limited testing, facilities or wit'n other 
probleiTLs in producing data. In tbis research (Kducat ional Testing Service, 
196:^), the median of the average class sizes for tne curricula was 69, but 
these average class si:aBS ranged down to 2 3 and 36, with an administratively 
enforced lov/er bound of 20. In a later recons titution of the curricula 
.^!;roups these prc^blems were alleviated somewhat, thouf^h for sotiK?, new curric- 
ula t^roups ii: was necessary to use data from schools which could supply as 
few as 25 cases. 

The problem of few cases at many locations arises also in research on 
black students. Cleary (I'^fvS) and subsequently Temp (1971) have encountered 
this problem in corinection with the study of the validity of the Scholastic 
Aptitude Test when used for blacks and whites at racially integrated schools. 
The study was undertaken, in part, to determine whether one should use the 
s ime prediction system, for forecasting grade point averages for blacks as lor 
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whitss, or, in effect, whether the two regression lines are the same. To 
conduct the study, Cleary located schools with sufficient numbers of cases 
so that a school-hy-Mchool approach could h. used; Temp's study followed 
the Cleary approach. In the opinion of the author, the Cleary approach is 
an excellent one, hut whether it leads to results that are applicable to 
most schools, or perhaps to most blacks is open to question. ^f a study is 
limited to schools which enroll many blacks, then one has not observed 
schools where there are few blacks, and in other phenomena of racial mixture 
the nature of outcomes to be observed may depend on that mixture. A study 
of a large number of schools virh few blacks remains of interest. 

The problem of locating minority groups becomes even more difficult at 
a more selective level of education. In an as yet unpublished report, 
Schrader and Pitcher (1972) compared the regression functions for blacks 
and whites in American law schools. Of the five schools from which suffi- 
cient data were aviil/ihle for this study, one had data for only AA blacks 
and the other for only 31; these cases were accumulated only by combining 
data across three and two years, respectively. In another unpublished 
study, the author (Boldt, 1971) reported a study of the validity of the 
Admissions Test fo/ Craduate Sc'nools of Business in which the group sizes 
-'d from 7, 10, anrl 12 up to U . In this latter study, a large number 
of schools had been approached for data, and six thought they had sizable 
groups. UTien it came to produciuK usable cases, however, the very small 
groups listed above were all that were forthcoming. 

Even without the complication of locating minority group students, 
the problem of con(hict ing validity r(M;earch on graduate education has been 
lont; exacerbated by a deartli of usable data. Summarizing attempts to do 



EKLC 



vaiidi.ly work fro::: 1^52 to 19b7 (^n the (irac'.iiatc- Kecj^rd I-xaminations r(;RK\ 
L.innhoim (1968) reportoci 22 stuciies in which performance prediction research 
was done and the sarin le sizes bv department art-^ ?>ive'n. Tn these studies the 
comhii^ation of dat:;i hv instivntioi^ v;.'is a'.*:^ ! ;] : , b'~\l vLthin a department it 
was s ome t iines nec e s s a ry to a c c umu la t e d a t a ov s e ve r a 1 yea rs , Th e aver a p,e 
group size per departmt.>nt in these studies v;as 31, the smallest being 7 and 
the larfZGSt being 185. Bv far the largest of these groups were from the 
education area, tlie group of 185 being in secondary education, an area where 
the c<v,irse work and majin's are nc^t interchangeable because different academic 
areas are involved. Indeed, as far as numbers of people are concerned, 
education is in a favorable position since graduate work is required for 
promotion or certification in many areas. But education also has many 
specialties with different requirem.ents and very different course work. It 
is suspected that pooling people for a study because tiiey are all in an 
educir'.tion department may not be appropriate and, in any case, education 
departments are not representative of the rest of the graduate world. If the 
nine studies reported by Lannholm (1968) which deal with education departments 
are removed, the average size of the rem.aining group is 37 per department, the 
smallest still being 7 and the largest being 96. In 10 of these departments, 
50 '^>r fewer cases rire involved. 

Lannholm, Marc(5, & Schrader (1968), being aware of the sample size 
problem, attempted to accumailate graciuate school data by inviting 32 depart- 
ments in 15 different universities to participate in their research. As a 
result, data were received from 21 departments of 10 universities, with 
sample sizes ranging from 8 to 116. The average size of the groups by 
department was A 5 even though these authors made a good attempt to get 
better and more numerous data than had been made by other authors. 



LanniK^lm c-(vuiniK.'cl iii .:i:ri'noii^ay of valioitv ^;lu;iirs v; i t h iiis 1^^;; 
sununary of ^:RK validity sLudi.'s fmn I9h'. [^^70. in ihv.^v sttuii.-s tlu- 
education dt-pa r tru'u ls r^nlimu-d to provide- si:'-ihl*' rrpi.rv. of (^asos, v i 1 , i i 
an average of I ID oaso^, liiou^h .;roup si-.-s as low as i: and J7 v.vrr reported. 
In departments otiior than eduoation, the- av.-raue numbor of rases roportoH was 
63 with a range fr(MTi ^ to U7--^ir shcuild b.,' mrntioiird that mxo of : ir.- scdiooJs 
witii a >^ronp size of 1^7 had acrunulaU'd it. data ovrr a p.-ricul of ]] x^^ars „ 
Clearlv, lieroir moasnres ar^- iU'e*doti to prtw.inrr a stu(l\-. 

(Ce ntral Predict icMt App roaches 

Kven though the pmhlnns of the ava i ] ih i 1 i t v of conipa:-a!Me data r-. well 
kno./n to researchers In the so^/ial sciences, the coi-monlv available rep.res- 
SLon models contain no provision for Inclusion of sets of criterion data that 
do not lie on the same scale. In addition, there are problems where sets of 
predictor data may not lie on the same scale. For example, the Rrade point 
.ivt-raKO at one underj^raduato institution may not be comparable to that from 
anotlier sendint; institution for predictinR performance at the graduate level. 
:-.oo:n and Peters ( ]Q6]) were amons; the earlv resea chers to investigate the 
prohlen of e,rad.' adjustment wi t!i verv sien.ificant results, tiiout-h results have 
not hehi uD in later studies. Tur-ker has nrmtioni^d certain teehnic-al 

problems th.at mi^ht account for this and developed n num.be r of formal models 
for central prediction. Thcst- models were lieve loped in the context of pre- 
diction at the undergraduate level usinR empirically adjusted h i Rh school 
p.rade point averages as well as empirically developed adjustments for the 
p.rade scales of the recei vinp, ' coll op,es . Tucker's models are least s.|.-.res 
m(Hlois, whereas those discussed by Potthoff (l')f,/,) in a paper sponsored hv 
the Kducati ona I TestinR Service (ETS) are models usinp; maximum likeliliood 



estimtion under tlu- .■onimonly made assnmot iotis of joint normality. Basluiw 
(1.965,1, 1965b) has reported central prediction models that arc formally 
identirai to that of Tuc:k..-r, liu.u>;h with a soniewlial differoni computational 
scheme. Linn (1966) in surveying research on Rrade adjustment in the sending 
institutions has supplied a more detailed discussion .)f these problems and 
rnoclo 1 s . 

Tucker's paper includes a prediction model which is responsive to an 
even nu.rc r.eneral fi^rmulation of the prohlem of prediction in allowing, in 
addition to adjustmeuts to grades from the sending and receiving institutions, 
for differences in the types of institutions involved, e.g., an engineering 
school as opposed to an institution concerned primarily with liberal arts. 
-Hie research probLeni to which the present paper is addressed is quite a bit 
more determined t ha.n Mie problem to which Tucker's predictive model is 
nddress<-d because onlv one kind of sending institution will be studied in a 
particular solution and no a.ljustment of grades from sending institiutions 
will be contemplated. Never tlie 1 ess , the least squares approach applied here 
is in the spirit of 'fucker's predictive model where the residuals whose 
squared sum is n,inimi?:ed contain no adjustment on the criterion. This is 
opposed to Pntthoff's maximum likelihood solution which determines a trans- 
formation on the rriteri.m scores; i.e., the grades at the receiving institu- 
tions. The ie.-ist squares solution used ^n the present paper is described in 
Appendix A nn.l was originally developed bv the author for use in the Cannon 
et nl. (L9hf,) stu.iv an<l was subsequently used by Carlson (.967) and in the 
•development of the CC1> hatterv (ilTS , 1968). Appendix R contains a least 
squares solution that adjusts the .-riterion scores and, aJ tlun,g:, It. is not 
.I'M desirable fron .i prediction point of view for that rea-^on, it has cei'tain 
computational advantages tiiat would be useful in ^.cst selection procedures. 



Bayesian Approach. 

The approach taken by some of the r-esearchers in central piediction is 
to etain an assumptic;n of structure similar to th.e linear form used in 
multiple regression but to allow additional linear adjustments that 
account for the source of the particular data set. Thus, the large body of 
data is summarized in relatively few pirameters, though mere than would be 
rec[uired by a single regression model and less than would be used by the 
JitLing of a whole set of individual constants to every data set. But the 
previously clisi:uss:>(i central prediction procedures have no way to take 
advantage o[ partial, or vague, information available prior to the estima- 
tion study. For example, if one were to estimate the regression of college 
grades on ACi': scores, one could examii e results from a variety of studies that 
would indicate something about the likely range of the coefficients to be 
found. Reasonable ranges for the means and variances could be set up as 
well as for the correlation coefficients. Surely, such a tudy would not 
be the first of its kind with entirely new knowledge being made available, 
but it would be at least a partial affirmation of existing knowledge though 
applied in a slightly new context. I"/liere several schools are involved, one 
would want to incorporate the notion that they are more or less similar. 
One would certainly not want to proceed under the assumption that all schools 

are uniquely different, conceivably, and that no prior information is in 
2 

e x is tence . *" 

Being aware of these problems througli discussion with M. R. Novick, 
Lindley has developed a Bayesian approach to the type of problem of interest 
in the current study. indetui, a series of Kducational Testing Service 
Research Bulletins (Jackson, Novick, & Thayer, 1970; Lindley, 1969a, 1969b, 
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l^TOa, j.^TOb) :iave ^locllne:■ lol' tlio Jcvelopnent nnd discussed api'^licit ions of 
the Bayesian npproncd-i to proMens In educational T>redic^'ion and p.uidance. 
LindleyVs approach has tiic incorporation of vague prior information, allows 
for local differerices in rcerossio:i cv^uatic^ns, and produces a method of 
proper balancing of sparse local data attains t data f ron. the entile set of 
schools under study. The estimcition procedures use a likelihood fimction 
weighted by a prior distribution, hence are more sinilar to Potthoff's (196A) 
methods than to least squares. Conipared to that of Potthoff, Lindley's 
method W'HiLd clearly seem to be the method of ciioice, since it employs a 
predictive model in the sense of Tucker and because it incorporates the 
desirable Bayesian features mentioned above. 

Lindley's model (1970a, 1970b) incorporates the following: the like- 
lihood is characterized by grades and predicted grades for which linear 
regression holds and whose discrepancies are normally and independently 
distributed; coefficients of linear regression have independent priors 
across colleges but within colleges have joint normal priors v/hich are, for 
a college, characterized by a vector of means and a dispersion matrix which 
is exchangeable with the veccors of means and dispersion matrices of other 
colleges; the priors for the exchangeable vectors of means and dispersion 
matrices are uniform and Wisiiar t , "respec t i vel y ^ in form; the dispersion 
parameters frorr the likeh.ood are excliangeable and have independent inverse 
chi-square priors wliose central tendency parameter has a prior wh.ich is chi- 
square in form. 

Nee d for Comparative Experiment 

The :ist of assumptions above is ratlier 1 on>: and narts of it mav he unten 
able-: the normal it v of errors of (b) is clearlv incorrect because all of the 

1 ^ 



v.ru-.hlcs are hoinuied, becaus... even if college grades were no mall y distrib- 
iiced, Che test scores are not; and because ine hnmoscedas c ic i t y implied by th. 
Independence assumption is incorrect. Furthe r-nnre , the assumption of nnlfor.n 
distribution en tne r.eans of (c) is nuite unreasonable considering the rather 
narrow ranee that can be involved. In fact one could probably reject any 
assumption cne could test, given enough data, and vet it is not proposed to 
abandoP_ the approa-.h. Rather, the assumptions, together with the estimation 
nro,...<iure can be used for data for a back sample, and the prediction system 
ran be tested in a rro.s sampK. as is often done to observe the effects of 
shrink due. to capitalisation on chance in the back sample. If one system 
works better than another, it is better even if arrived at through some 
quest ionab le assumpt tons . 

Novick, Jackson, ITiayer, & . ;ie (1972) condt.cted suci, an experiment 
using data from the liasic Research Service of the American College Testing 
(ACT) Program. Their comparison of the Bayes prediction system was m.ade 
with reference to the standard least squares procedure where estimates are 
made ac eacli location and independently from the others. In this study, 22 
schools were chosen, and data from each provided a back and a cross sample. 
The back sample, which was used to develop regression parameters, was 
collected in 19r,.H; the cross sample, used to evaluate the prediction system, 
was collected in 19bO from the same schools. The back sample sizes averaged 

with a high of 719 and a low of HI; these sizes are considerably in 
excess of the troublesome ones wi ti, which one often deals, and one might 
expect that shrink effects usually observed in tiie cross sample would be at 
a minimum. In lact, the drop in validitv averaged over schools is very small, 
going from .SI to .47. This suggests that capitalization on chance Is not a 



very potent factor diid leads one to exj^n.-rt thai the u-e of prior information 
afforded in the Rayosian systetn would no^ lit^' crucial. 

To compare the Bayesian and least squares results, Novick o_t a_i. (1972) 
used four indices. Kach of these indices us^s a ni -dieted ^;core .and an 
actual score.. The predicted score uses a prediction function whose- parameters 
are computed in a back sample and whose argument^ are the four variables of 
the American College Test; thet.e is a different/ set .of parameters ' for each _ 
school, and Novick e£ aj^. (1972) presented the four indices for each college. 

ITic indices were (a) the familiar Jaean square error (MSE), which is the 

, / 

average of the squared differences l>:etween the predicted grade and the 

/ ■ ' ' 

observed grade, (h) 'the average, absolute error (AE) , which is the average 

of the absolute valM^s of the difference b-tween the predicted grade and 

the observed grade; (c) the :/ero-one .loss (ZOL) , which is the average of a 

variable that is zero if the prediction is within half a standard deviation 

of the actual grade on the observed grade scale and one other^-^ise; and 

(d) the correlation between the observed sc^/es and the predicted scores 

(COR). ■ The averages were of the indices, giving least squares results tirst 
and Bayes second, (a) . 56 ' and .55 for MSE, (b) .58 and .58 for AE, *(c) .56 

and .56 for ZOL, and'^d.^ .47 and .48 for COR. Although the sample sizes 
'are not given for the l^V samples, the^a^athor seriously cfoubts that they 

are large enough to detect significaye for t^he" small differences shown; 

if they are different, the difference is certainly not of puch practical 

in teres t . . 

i 

In snialler samples:, the effects of capitalization on chance are more 

* 

marked, and one would expect to observe an e;..iancement of the value of the 
Bayes approadi over that of .l:he usual regression approach in snail samples. 



To observe whether such might be the ease, Novick ej^ a]_. (1972) drew a 25% 
random sample of the 1968 data for use as a back sample and crossed the 
results into the 1969 data. ITie avei^a^*^ values of the resulting goodness of 
fit indices are as follows, giving the least squares results first and the 
Bayesian results second: (a) .62 and .56 for MSE, (b) .61 and .59 for AE, 
(c) .59 and .56 for ZOL, and (d) .42 and .47 for COR. A further advantage 
of the Bayes approach was that it did not yield negative regression coeffi- 
cients as did the least squares method; except in special cases one does not 
usually accept negative coefficients in a system in this context. The aver- 
age back sample size was 61, with a range from 26 to 184, which is not as 
small as sometimes occurs but is small enough to indicate some superiority 
of the Bayesian method to which th.e indices above testify. -Novick al. 
(1972) point out that relatively more gain would he made with even smaller 
divisions, such as splitting the college groups by sex, 

J 

Leas t vSq uares vs . Bayes 

Although the Bayes approach due to Lindley appears ^prjomisirg in the 
work of Novick e_^ a_l. (1972) as described above, the question can be raised 
as to whether the superiority of the system as compared to the least squares 
system is bi/cause liayes is better than least squares or whether a poor 
least squares approach was used. Most researchers would not use a standard 
regrej^sion in a situation like that of the 25/': ACT sample. Indeed, Novick 
and .Icickson (1974), using (probably different but) exchangeable regression 
ctX' I f i c ients across schools, but \<\x\\ different intercepts at ea'cli school 
recently reported that the mean squrfr^- error in a cross sample approached 
that of the B<ivesian method. In Cannon al. (1966), Carlson (1967), the 
CCP battery (1968), and Boldt (1971), the le:-St squares method of Appendix A 



was used i p. orJor Vo t over the 'problem of l\\o si'iall number of c\isos :\i 
each lorat ion. In fh Ls moihod, nuiTil)t*r o\ pitrametc^rs lo be fitted is 

equal to tiie iuir;ber of predictors U^^.s one, plus twic^e the number of schools; 
with a lar^^,e nunber of schools tb.L^ n.!.! alive W(/i^:hts for tlie predictors 
would be f^reatlv overde to rmined , and one would expect the system to be quite 
stable w' th few cases per school. Since a least squares approacli wliicli 
featiires more overde termi na t ion would bo more appropriate and would seemingly 
be competitive with the Hayes approacli, a study comparing the tv;o was under- 
taken. 

Sample 

In spite of the researclier Vs ineluctable confrontation with ^he sample 
size problem when attempting research at the graduate school level, it was 
hoped tliat th ? prc\senr study L'ould be accomplished usir^ data from that 
source. The letter in Appendix on CRE Board letterhead was sent to the 
p,raduate deans of 05 American graduate- scliools. The appropriate second 
page was included, depending on whether the school would be asked to contrib- 
ute data on psychology or economics graduate students, or on both. Eighty- 
one departments of psychology and departments of economics were approached; 
these departments were listed by Lannholm (1967) as requiring the ORE apti- 
tude examinations as well as the advanced examination in the field of 
specialization. Enclosed v/itli the letters to the deans were copies of the 
CRE Validity Study (Questionnaire and letters to the appropriate department 
chairmen. Hiese ay) preaches were made in November 1970; four months later a 
follow-up letter, a second copy of the questionnaire, and a copy of the 
oripjnal letter to d«q)artnient r-halrmen were sent to eacli nonrcsponcli ng 

de.p;irtment chairman. llieso sol Lc i ta t Lons emphasized intended convenience in 
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siipplying daL.'i nnd sc^liit i(m lo t lio r.c^mmcm prc^blems faced in doinp, research 
at the. graduat:e level. In particular, ilie departments were urged to respond 
even though they might have few c.-^ses to supply. Fifty-seven or 70% of the 
psychology departments and 36 or 67% of the economics departments eventually 
replied. Of those v/ho responded, 32 psychology departments and 15 economics 

departments eithier indicated an unwillingness or, refusal to participate or 

i 

staled that they would not be able to provide usetiii^data . Reasons included 
the fact that no data were available for Ph.D.^s, that no graduate grades 
would be made available because of confidentiality, that ORE score require- 
ments were not actually enforced and were available on only a few people, 
and that no advanced ter- 1 scores would be available. Also, five economics 
and nine [^sychcslcgy departments indicated that to search the graduate files 
woui.d work a hardship on their clerical staff. Two economics and five psychol- 
ogy departmeHLs gave hoi. h IcLnds of reasons of the total who responded, 25 
psychology departments and 21 economics departments were usable in the study. 
Of these, 16 of tht: nsyciiology departments and eight of the economics 
departments actually computed routinely a graduate grade point average. 

The eight largest of the 16 psychology departments were contacted. 
Based on r.^sponse to the CRF. Validity Study Questionnaire, data for approxi- 
mately 60 Ph.n.'s and 60 master's graduates\should have been receiv^^cl, rang- 
ing fron. 40 to 80 for the Pli.D.'sj, and from 15 to 105 for the master's. Of 
these, onlv four supplied any usable data on students for whom grade point 
averages were also supplied, and these gave 4, 21, 8, and 7 cases at the 
IMi.l). level ai'.d iS, 30, 60, and 17 at the master's level. Tiiese four schools 
abio sui);-)! ie<l (int:a for 31, 7, 24, and 4 students v;ho ^';ot bc^th t hc^ master's 
and the I'ii.i). 'If-t'^rees . C^ne scliool provided data for two doctoral candidates; 
one sciiooL ran into d i f i icu 1 1 i(\s whc^n it came to actually accomplishing the 
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clerical v/ork, and at the lasl school i. f proved Lippossible to call .■ ; get a 
return call from the appropriate contact. 

Three of the largest economics departments were contacted based on 
responses to the (iRH Validity Study Questionnaire data for approxiTrTa tely 
AS, 40, and 30 Ph.D.'s and 7S, 70, and 60 master's candidates.-^ Actua\py, 
no data were received; one scliool would not respond to telephone calls, one 
required financial support for tlic clerical work, and one reported it was 
'plodding/' 

Clearly, if the ORE da<:a wore to stand on their own, the project was 
in trouble; the remaining schools tliat provided graduate grade ^oint averages 
had only a few candidates and witli similar attrition .^ould not be helpful. 



and the schools for whom transcript analysis would be require^ were also 



been one with the main emphasis on the validity of the ORE, one would have 
seriously considered a course of action in which a large number of graduate 
schools with only a few students would be solicited for data; in doing so, 
one would be in the k-;.nd of situation for wliich the models under discussion 
are designed. Sincj he point of the piresent project is methodological, a 
better source of dataV- required- 

Analysis of VSS Data 

The files of the CEEB Validity Study Service (VSS) were examined to 
find data in whicli colleges h-^d, on two successive occasions, participated 
in a validity study using comparable groups on both occasions. Data for 
12 such colleges were obtained, with data for both males and females. The 
back sample consisted of 25 cases pulled at random (Tauswortlie, 19h5; 
V/lilttlesey , 1968) from data collected in the first year of participation by 




schools for whom few returns ^ould be forecasted. 



If the present study had 



i. .. 
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the colleges in the. VSS. Since there were samples for i^ales and females for 
all but two schooly, the randomization was done using a list of males sepa- 
rately, a list of females separately, and then a merged list with no control 
on the composition by sex. Two of the schools provided data for females only; 
hence, the back sample for the analysis of "combined" data consists of an 
independent random sample from the same group used in the analysis for 
female cases. The average cross sample iizes are 13A, 173, and 285, with 
lows of 48, 74, and 74, and highs of 204, 353, and 555 ■ for the male, female, 
and combined groups respectively. The low for the female group is equal to 
the low for the combined group because the sample with 74 cases was not part 
of the male group, but was taken from one of the two colleges for which no 
data for males were available. Tables 1, 2, and 3 contain summary statistics 
for the schools involved for males, females, and combined groups, respectively. 

As with ihe study of Novick _et al. (1972), this analysis is concerned 
with the relations betv^;eep a predicted grade calculated using a prediction 
function whose parameters are developed in the back sample and whose argu- 
ments are the test scores (V and M) and a grade point average (LfR) observed 
in the cross sample. For each student in the cross sample, there is a pre- 
dicted and an observed grade point average. The correlation between these 
is COR, and the difference between these Is called the "fesidual." These 
residuals are used to compute AE and ZOL, as defined earlier. In addition, 
an average residual (AR) , the. variance of the residuals (VR) , ^.afid^ie low 
rrnd high residuals will be reported. All of these indices of goodness of 
fit of the predic:tion systems w L^l I b^,; reported by school for both the 
Bayesian and leas: squares systems .is applied to the data of »nThles, females, 
and the (^ombin'fl group. will also he presented for various combinations 
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of tlie SchoLastL.; AptLUido Test Verbal (V) urn] Mathemat i (M) and the high 
school record (.M). No adjustment o\ thv W Is Included In the models con- 
sidered. 

Tables 4, 5, and 6 sliow the results for the index COR^ for males, 
females, and the combined sample. Nntic:e that in these tables the Bayes and 
least scjiiares predictions are about equally ^^ood with the exception of the 
negacive entries. For these entries, the vagaries of the back sample are 
such tfiat a negative mul t iplicative constant is needed to put the predictions 
on the grade point scale for the particular college because the predictions 
correlate negatively with the grade point average. For example, at college G 
the V scores of the males correlated -.01; therefore, the sign of that. pre- 
dictor will be reversed. But even if the back sample results indicate it, 
one does not accept that the correlation of V with grade point average is 
negative at this college or probably at any otl>er unless the grade point 
scale is inverted; in practice, the negativ^^ weight would simply not be used 
on a priori f^rounds . The auliior is awai .1 this sort of reversal, hav..ng 
encountered it in the data for black students at the graduate schools of 
business (Boldt, 1971), and feels that, where such sign reversals are found, 
a reasonable practice to adopt would be one in which the absolute value of 
the multiplicative constant developed in the back sample is used in the cross 
sample together with an additive constant which is adjusted (as described in 
Appendix A) to account for the change in sign of the multiplicative constant. 
The additive constant would not affect Tables 4, 5, and 6 since the correla- 
tion coefficient is invariant undd-r an additive transformation, but the sign 
would be reversed and one can see that, with the sign reversal the Bayesian 
and least squares systems are about equally good. The author wants to stress 
that this change in the sign was not a change suggested by the data but was 
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intondod tor use prior t:o tho collection of any datn in connection with 
this study. A further consideration of tlie sign change appears in the 
Discuss Lori and Conclusion section of this Bulletin. 

Tables 7, 8, and 9 contain the average values of the residuals (AR) 
found in the cross samples. Though the entries where the negative multi- 
plicative constant occurred are footnoted in these tables, the values 
entered are calculated with the adjustments referred to in the paragraph 
above. Examination of Tables 7, 8, and 9 reveals a slightly larger number 
of cells in which the average residual from the Bayes system .s smaller 
than the average residual from the least square!; system. These errors are 
not trivial in all cases and are highly responsive to fluctuations in the 
additive constant, a mafter which will be referred to later. 

^•'^ ^" '''''^^ ' ^' ■ ■ '^"'J Tables 10 through 18 contain entries which 
are corrected for the negative correlations of predictor sets with the 
criterion in the back sample. However, Tables 10, 11, and 12 contain as 
entries the variances of the residuals, VR. Like the data in Tables 7, 8, 
and 9, the advantage seems to be slightly to the Bayes system in terms of 
the frequency of cells in which the VR is smaller for Bayes than for least 
squares. On examination it can be seen that cells which are footnoted c 
are not necessarily the ones in which the least squares system fits less 
well than the Bayes; the use of the absolute value of the :nult iplicat ive 
constant seems to have been reasonably successful. It should be noted that 
the index VR does not provide information about the additive part of tiie 
transformation that puts the predictions on the college scale, since VR is 
invari/mt under linear Lrnns format i on of tlie variable;;. 
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Tahlcs r3, LA, and .1 ^) contain Iho vahu's n\' 7.01.^ and Tables 16, 17, and 
13 contain values of AK, No pa rL i 1 a r atlvanlaK*-' for Hayos sysLi'in L,s 

observed using Iho ZOL measure. 

Tables 19, 20, and 21 contain tlu' parameters of the Hayes and least 
squ^ires solutions. Note that in contrast tc^ the studv by Novick e t a 1 . , 
some of the rep;ression coef f icien ts are ne^',at i ve in the Bayes system. Table 
22 contains validities of the predictions using the Bayes solutions shown in 
Tables 19 through 21, as well as the average Bayes weights and the least 
squares weights. Table 22 also contains the validities that would be 
obtained if the Bayes solution were obtained merely by reversing the sign 
where the weights are negative. Note that a solution with positive weights 
is better than one with negative weights even for a cross sample on the same 
school from which the negative back sample weights were derived. This 
reversal of weights is treated further in the Discussion and Conclusions 
section of this bulletin. 

Analysis of GRE Data 

Despite the scarcity of data from the graduate schools, a back sample 
analysis of the data was conducted. In interpreting the results of this 
analysis, the reader should keep in mind that the returns are highly selec- 
tive in the sense that the ability to supply data for the study classifies 
the participating ins ti^tut ions as atypical. Data were received from four 
schools and cases are identified as receiving a doctorate or as terminating 
with a master's degree, allov;ing eight classifications of students. Table 23 
contains descriptive data by school for these groups and for the combined 
educational groups. Prediction analyses were conducted for each school using 
all eight classifications (Combined 8), the terminal master's only (Master's), 



the doctoraLc on.l.y (Doctorate), ;u\d the school, ignoring the degree received 
(Combined 4). Validity coefficienLS for these groups are presented for'vari- 
ous combinations of system predictors in Table 24. These system coefficients 
squared give the percent of variance accounted for over and above the group 
means by the predictors. The computation of tlie coefficients for the least 
squares system is described in Appendix A, equation (11), and for the 
Bayesian system the system coefficients are calculated by combining correla- 
tions of weighted sums, the weights being the regression parameters esti- 
mated using the Hayes approacli. Examination of Table 24 shows that the 
highest system validity coefficients are those in which the most parameters 
are fitted. For example, parameters are added as predictors, and one may 
note from Table 24 that the coefficients increase as one moves doi-m the table. 
Also, the Bayesian system fits more parameters than does the least squares, 
especially as predictors are added. One may also note that the discrepancy 
between the least squares validities and the Bayesian validities increases 
as predictors are added; as each predictor is added, one parameter is added 
to the leai-t squares system, hut to the Bayesian system as many parameters 
are added as there are schools (four in this case). ThereTore, the trends 
noted in Table 24 may be the results of capitalization on chaace, rather than 
reflections of reliable trends in the data. As a check. Table 25, which 
gives system validity coefficients in the back sample of- CEEB Validity Study... 
Service data, is offered for comparison with Tables 4, 5, and 6. It can be 
seen t^iat the least squares and Bayesian coefficients are about the .^ame for 
larger numbers of predictors. Bayesian coefficients are smaller for the 
single predictor case (the single predictor case involves more constraint for 
Bayes than for least squares). In Table 25 the validity of the VI1H composite 



is :\hn\[l that, i^f !! aliJius ami in T/ib A. '> , ri-l t lu' sch:>ol c.oc f f i c i cn t: s arc 
abouL Lhe same as in Table .! > . Sinilarl.v in 'ra>^ 1 e 2A rhe least squares valid- 
ity of VQPU is abf)iit that ul L' alone; thus, if a sinalarity v;ith CHEh Validity 
Study Service data holds, c>nv may cxixn^t ,svstem cross-validit.ies of about .3 
to .35 usin^* the least squares pretl i c I o r.'- . An important question about the 
data in I'able Is whether the lar^;e validity coefficients for the four 
pretlictor Bayes systems would hold up under cross-validation. It is the 
author's (impression that they would not since the Bayesian system is adding 
only about 18 correlation points for an increase of almost 12 parameters. 
It is, of course, a matter for additional study whether the Bayesian system 
produces results that would stand up-wLth such a paucity of data, but in the 
author's opinion it would be extremely optimistic to accept the validities 
in the range of .5 from tlie bottom line of Taole 24. It may be reasonable 
to expect, however, validities in the mid- th i r t ies , provided tiie undergradu- 
ate average is included. 

A further reason to question tlie validity of the regression composites 
in a cross sample comes from examination of the regression coefficients. In 
the Bayesian solution for the Combined 8 groupings, only for U alone and for 
Q and U together were the regression coefficients positive for all schools. 
For the doctorate groups, the regression coefficients were positive for all 
schools only when U was used alone. For the combined grouping, regression 
coefririents were all positive only when using P or U alone, Q and U together, 
and P and U together. This means that, for all other combinations of varia- 
bles, one would be using negative regression weights in a new sample, and 
these combinations include, for example, the four variable predictor set that 
yielded the back sample system coefficient of .52 in Table 24. The author 
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docs net ac:cei)L tlu* t'Dnc 1 us i dh llirit: negative cnef f ic i.ents are correct for 

other samples but considers tiien an accident of these data. 

Discussion and Conclusio ns 

ProbleiTia in data collectiop have been discussed in some detail, where 
only a few cases are available wi/.h criterion data on a coinmon scale, or at 
least on a scale which is known to be common. Minority group research is 
Kiven as an example of a situation in which such problems arise, and the 
I)roblems increase when grachiate student populations are involved. The dif- 
ficulties (mcountered in th.is study definitely make it clear that improvinp; 
the state of knowledge In these very important areas will require the 
cooperation and effort and even some trouble on the part of many institutions 
that could provide data. To j^ain the cooperation of institutions probably 
requires convincing them that the solution of the problem to which the study 
is addressed is oni^ in which tliey have an interest. Unfortunately, since 
the methodology under study is one designed to deal with small samples at 
many places, the sample size itself may preclude the development of a per- 
ception by the [)arent institution that .the group for which the data are 
collected constitute in themselves a cause for concern, or a reason to take 
the trouble. It is hoped that some effective approach to the data collection 
problem will be found in the context of graduate education. 

The pr:?sent sf-dy asks whether the Bayesian system and estimation 
procedures, used by Novick ,o_t £l . (1^72), would prove superior to the least 
squares system used here (Appendix A) whicii, in contrast to the usual regres- 
sion system, allows pof^ling of data across colleges. It was found that, if 
negative multiplicative parameters are developed using the least squares 
system are converted to positive and if the additive constan^s are adjusted 



acco ril j.a>':l.y , l.lie 1i\isl ^".(|uarL'S ';vv, ttT.i .ind lIh' Havi-; syr^tci:! nro ixhoul eciunll.y 
S';oiul in a c ri^ss-va 1 i ila t i on stiiJv luaul i t i on of first-vt'ar collGr,e erndes . 
Five; in^iices Vv'i.Tc- usod to iiulit\itt' fit of t\\c proclictii^n In the cross 

sample: tliu validity (COR), ilu avrrar.e resul\ial (AR) , the variance of tlie 
residuals (VR) , the zoro-ono loss C'Oh) , and the averaj;e error (AK) r The 
differeni'e between estimation na^thods pr(K]uced very little variation in those 
f i y,u r es o f me r i t . 

The least squares estim«ates were subjected to si^n reversals whore 
negative multiplicative parami'ters occurred (the entries wliere this happened 
are noted in Tables 4, 5 and 6). (Clearly, had tlie reversal in sign not 
occurred, the values of COR would have been negative — prediction would tiavo 
been backward. Other indices of fit in the back sample with the exception 
of AR would have suffered as v;ell. But in the tables the Bayes and least 
scjuares results appear to be about equally as good. Therefore, be f ore tlie 
sign change Bayes was i'>etter. 

Realizing that tlie treatment c^f the two methods had not been entirely 
symmetric, the author examined the Bayesian regression coefficients to see 
if some of them were ne;?ative and might be changed. Some were indeed found, 
their signs reversed, and the results of that reversal are presented in 
Table 22, Tt can he seen that reversal of the negative signs in the Bayes 
forraulae improved the Bayes predictions, also, as tlie author expected. 
Symmetric treatment, treatment which leaves the signs alone in both systems 
or treatment whicli changes the signs in both systems, seems to leave the 
Bayes system witli a slight advantage. 

Readers will undoubtedly differ on whether the sign changes arc^ 
acceptable. T]w autiuM" just, if ies tl .nn on several grounds. First, experience 
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sV.(n;.s tiiat wlu:n apt iliulf aiu! :',r .11 r 11;;^! in pr'ndic^t later graciu.-; , the 

rcp.riission woi);hts ar(.' ovt^t^wlu^lniinj; ly i t i vc» v.'Iumi samp](.\q ari? adecjuate. 

Second, Iht- bac^k sam|)l.L's aru siivil). Tfn'rd, clrinR(\s wore pjossible v;ithout 

pos I hoc . re 1 Lance uri cros:; sampU^ criterion data, v^.ce.'-.s airy characteristic 

of any acceptable e.s i ma t Lon procedure. Fourth, positive weights make better 

policy senst*.. I'liest^ points are amplifiecl below. 

Tiie least squares system used here is particularly prone to sign change 

errors by collex^,e as examination of Appendix A reveals. There it can be 

seen that the prediction formula is of the form Az + B where z is a 

linear function of the predictors; the parameters of z do not depend on 

the particular college. Iherefore , the coefficients of Az + B are 

estimated by tlie forr^iula for regression of grades on z at 

particular cc^ lieges. I'liese weights are based on only 25 cases so it 

(^ne obtains a ne^'it^ive mu 1. 1 i pH cat ive constant for one college and if one 

must reconunend a pri'diction formula for that college, one has to believe 

that aptitudes and grades predict backwards at that college to recommend 

the use of the negative value (in the present data all the scales are 

arranged so that *\good*' grade point averages are large ones). Therefore, 

in substantive terms a negative value for A means that increasing Verbal, 

Math, and undergratlua te, grade point average implies decreasing grade point 

/ 

averages. The author/declines to accept that the colleges here are that 
strange; t fie re fore the sign reversal or a [^^ero weight is indicated. Zero 
ml^ht be ccmis ide rtul as a solution l.^ecause one might think that a negative 
weight estimatt\s n ;'.ero; the zero migjit also be accepted as tlie estimate for 
a least squares objective fancti(Mi like that of Appendix A but wita the 
explicit constraint that multiplicative v;cights must be nonnegative; but i^ 



r,rncies witii ovun-'lif InlnK rroquiincv :iiu{ tiu* use nf ^ nuiltiiplicar i ve wcipjit 
zero v^niild linLl uiu^ lu r.rt)ss val i '.1 1 n' i l\s of y.oro — one- knows one can do boLtcr 

Accepting Lliaf p(^sitivc wel^^hls aro needed, wliaf should their magnitude 
be? rntiiLtion says that tiu^ varinnco of thc^ predictions should probably he 
somewhat less than that of the crlteri(m: as a basis for produc_in}:; that 
variance one night assume that the correlation coefficient was about right 
in size but wrong in sign. If so, the weight should be reversed in sign as^ 
was done with the results here. But other solutions in the form of othe^ 
least squares objective functions were sought; only one appeared worth 

checking empirically. In this alternative least squares metliod the- quantity 

/ 

w for a college was taken as proportional to tlie inverse of the multiplica- 
tive constant for that college thus ensuring the existence of a relative 
minimun for the objective function in the regir)n where the n.ul t iplicative 
constants for the colleges are all positive. Tn this case tlie es t ima t or "of 
tlie multiplicat ive constant for a college turns out to be the ratio of the 
standard deviation of the grades at the college to the standard deviation of 
the- ;^'s at the college; the estimator eauates the variance of the pre- 
dictions to that of the criterion. As one might expect ^rom the intuition 
mentioned above, this estimate does not work as well as the simple sign 
change that was actually used. Results obtained using the ratio of the 
standaTTd deviations were not tabled since the estimate is of no further 
interes t . 

The discussion above deals with the le;^st squares estimates. Bayes 
weights were also reversed to attain a symmetry of treatment of the two 
methods, since in the author's judgment positive weights would improve 



}• redic tit)n, in pan because the ovenvhelmin^. body of evici. -ncc indicates 
that such weights should be ' pos i t i vc , The other part of the reason for 
rejecting, negative weights can be appreciated if one supposes that a nega- 
tive weight for math was obtained at a college, and that that v/eight was 
incorporated Into the admissions pt)licy. 'I'hen one would almost certainly 
encounter a canditlate whose grades and vt^rbal scores were good but whose 
raach was so high it offset his grades and verbal ability and led to his 
rejection. To explain to the parents and other institutional officials 
that the high math score makes him lose the admissions race would seem to 
the author to be exceedingly difficult. Especially would this be so where 
the ^ights arise from Jimited quantities of data forthe college, a situa- 
tion for which the current methods are intended. For political reasons one 
would want to be absolut(^ly sure that the negative weight is correct, and 
this certainty would f)e ncuuied in a situation where the great bulk of evi- 
dence su);g,t>sf.s that i\ic welglits are positive and tliat the negative wci^^hts 
are a product of instability duo to tlie Jir.dted sample size. Thc^ author's 
judgment was ihr.l the negative weigjit.s are prc^hably not right, and that 
prediction wc^ild be improved if the signs were reversed. That proved to be 
the case. 

The sign reversals used in the present study could be applied in pre- 
diction contexts less familiar than that of tvUicational prediction where 
such well-known variables as V, M, i \\ are used. If one were using the 
^ least squares neth(nl, one would men^v reverse tlie signs of the few schools 
which seemed to work backwards from the rest. If one were using thc^ Bayes 
metliod, .one would reverse the sign nf the predictors wherc! only a few dif- 
fered in sign rrom thc^ others. (n either case, if more than onlv a few 



parameters showed il i t fereneu in siy^x\^ onv ml e^ht sus{^ec:t.- ihat i n>*^ t: 1 tut i ons 
are bein^ grouped v;hich should not be. UHiore such sign reversals are done 
in the least squares system, the additive constant Is determined by choosing 
a value so that the average back sample residual for a .sch.ool Is zero. This 
change is rather simple, but the change in additiv^e constant for the Bayes 
model is more complicated and iias vet to he workeci out . 

It is interesting to note that tlie arbitrary sign changes are much in 
agreement with that part of the philosophy of Bayesians which says that prior 
knowledge sliould reasonably be expected to influence the inferences one makes 
from sets of data. In this case one would prefer, of course, that the Bayes 
system be set up so that "the occurrenre of negative weights is unlikely or 
simply impossible. Rut I.indley and o er statisticians write for many 
applications and the generality of their methods would be limited if they 
included priors where negative coefficients were not possible. It is reason- 
able to suppt se that niensurement specialists might seek prior distributions 
which are r.ppropr ia te to their special context of application and that such 
changes would irnprove estimation in that context, even though the results 
would not b(^ .i^. gener.illy applicable as would Lindley's. Such change and 
improvement in the models for use in the data of interest to educational 
measurement would constitute progress in the science of the subject. 
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"In practice, even researcliors whp use classical statistical methodology 
use available prior information qui.:c often. For example, if a negative cor- 
relation of verbal test scores with college grades were found, the researcher 
would exar.ine the computation processes very closely, as he would the sample, 
probably until some reason were found to judge one or the other pathological* 
And his behavior is not entirely unreasonable because he "knows" that if a 
new sample were correctly dra^m and correctly analyzed, the resulting 
validity would be positive. He would behave similarly if the analysis pro- 
duced a validity of .95, though he might be somewhat less reticent to record 
the result. 

3 

For the single predictor cases, the least squares and the Bayes systems 
fit twice as many parameters as there are schools, but the Bayes parameters 
are interconnected throu^ the prior distribution. There being no intercon- 
nection of the least squares parameters, one may regard the least squares 
validities as being more subject to shrink in a cross sample. But as predictors 
are added, the Bayes system adds one parameter per school per predictor while 
the least squares system add^ only one parameter per predictor. Even though 
the Bayes parameters have some interconnection through the prior distribution, 
it seems to the author that for larger numbers of predictors the Bayes system 
must be more subject to shrink. 



4, . 

It IS commor to compar(' corrv];n ion jvi.ins tc^^the numbor of nnrnmeters 

' ■ ^ .cr vaiiujLv sjiTiiiK in a new 

sample is gruatily incron^od as tlic namb^T of parameters is increased. 
Although this pra-tire has grown up in a least squares context, it is sup- 
posed that, the same comparison would apply witli the Hayesian svstem. A gain 
of 18 corrdati HI points for 12 parameters is quite small from this point of 



V 1 ew . 



Table 1 



Summary Statistics on High School Record (H) , Undergraduate Record (UR) , 
CEEB Verbal (V), and Matli (M) for Undergraduate Male Cross Sample 



No. Cases 179 48 116 204 151 109 93 129 112 202 



!^leans 

51 30 57 52 47 60 47 43 39 50 

56 34 60 56 51 63 49 46 41 54 

H 3.02 2.57 3.10 3.05 2.62 3.13 2.84 2.49 1.94 2.91 

UR 2.54 2.03 2.52 2.80 2.20 2.05 2.31 2.01 1.80 2.53 

Standard ^ 2 



Deviations 



o 



V 969 11 999g98g9 I 

M 8 6 9 12 10 8 10 § 8 8 § 4 ^ 

H .44 .52 .59 .53 .62 .43 .40 ^ .56 .55 ^ .41 

UR .61 ,68 .70 .47 .62 .81 .58 .84 .69 .56 



Correlations 



V M 


.34 


.45 


.38 


.68 


.64 


.52 


.55 


.56 


.57 


.50 


V H 


-14 


.42 


.17 


.47 


.50 


.14 


.30 


.29 


.36 


.48 


V UR 


.39 


.46 


.31 


.15 


.40 


.20 


.34 


.33 


.41 


.36 


M H 


.32 


.22 


.40 


.49 


.59 


.20 


.37 


.33 


.28 


.45 


M UR 


.31 


.19 


.16 


.08 


.43 


.14 


.30 


.33 


.31 


. .32 


H UR 


.03 


.31 


.43 


.33 


.54 


.48 


.58 


.53 


.44 


.55 



Reported on one-tenth College Board scale . 



ERIC 



No. Cases 199 ^"^ 120 Iji 116 1^2 138 34/) 135 128 7^ 353 
Means 

>;.) 2^ V Sn 49 59 48 50 44 41 50 50 



M 



)2 11 V 55 4/ 56 
3.21 2.H 



47 49 45 41 ^.9 50 

;,;) ],];; 2.91 1.26 3.O6 2.78 2.95 2.64 3.04 3.20 

2.65 1.8' 2.; 5 2.98 2,4d 2.15 2.68 2.46 2.36 2.26 2.73 2.75 

Standard 
Deviation.s 

V 9 3 8 8 9 8 lU 8 8 10 8 9 



9 6 9 9 9 9 9 



H 


.4:i 


.49 


.40 


.3? 


.65 


.47 


.44 


.62 


.60 


.67 


.39 


.46 


UR 


.5Q 


.37 


.55 


.43 


.70 


.62 


.54 


.70 


.75 


.78 


.51 


.52 


rrelations 


























V M 


.49 


.43 


.50 


.47 


.59 


.47 


.59 


.52 


.65 


.53 


.50 


.55 


V \h 


.36 


,33 


.25 


.37 


.49 


.28 


.33 


.42 


.46 


.39 


.32 


.51 


V UR^ 


.43 


.50 


.54 


.24 


.53 


.27 


.57 


.40 


.56 


.55 


.40 


.52 


M H / 


.30 


.27 


.38 


.40 


.58 


.47 


.43 


.37 


.44 


.49 


.28 


.53 




.3/. 


.23 


.47 


,27 


.50 


.3 


.44 


.43 


.52 


■.52 


.40 


.46 


H IT. 


M 


.50 


.43 


.44 


.66 


.48 


.62 


.60 


.72 


.64 


.39 


.65 



Reported on one-tonLb College Board scale. 



3. 



',ble 3 



A B C D 
No. Cases 378 141 236 424 



Means 





51 


29 


57 


54 




54 


32 


58 


55 


H 


3.13 


2.61 


3.27 


3.22 


m 


2.60 


1.9) 


2.64 


2.90 



Standard 
Deviations 

V 9 6 9 10 

M 9 6 9 13 



H 


.44 


.50 


.53 


.49 


UR 


.60 


.61 


.64 


.46 


rreiacions 










V M 


.42 


.45- 


.43 


.59 


V K 


.23 


.35 


.20 


.46 


V UR 




.49 


.40 


.22 


MH 


.23 


.23 


.^0 


.42 


MUR 


.30 


.26 


.25 


.15 


H UR 


.66 


.41 


.46 


.41 



raJuate Cross Saiiipie. Both S^y.ts Cor.bined 



E 


_F 


G 


H 


I 


J 




^51 






284 


240 


48 


60 


48 




44 


40 


50 


60 


48 




46 


41 


2.75 


3.20 


2.97 




2.74 


2.31 


2.31 


2.10 


2.53 




2.20 


2.09 


9 


8 


9 


a 

1 

3 


9 


9 


9 


9 


9 




8 


8 


.65 


.46 


.44 


w. 

H 


.63 


.71 


.70 


.71 


.58 


in 


.81 


.76 


.58 


.47 


.56 




.60 


.54 


.50 


.21 


.32 




.38 


.38 


.47 


.23 


.46 




.45 


.50 


.51 


.31 


.35 




.34 


.33 


.40 


.23 


.31 




.40 


.41 


.61 


.48 


.63 




.65 


.60 



Reported on one-tenth College Board scale. 



Table 4 

Values of COW^ in tlie Cross Sonipje: Males 



Predictor Combination 



College 


System Type 


V 


M 


H 


. V,M 






V,M,H 


S amp 1 e 


A 


Bayeg 
L.S. 


.39 
.39 


.31 
. 31 


.63 
.63 


.43 
.43 


.60 
.69 


.62 
.64 


.69 
.69 


179 


B 


Bayes 
L.S. 


.46 

.46 


.19 

-.19' 


.31 

' . 31 


.44 
. 36 


.48 
. 3F 


.27 
. 33 


\ 40 
. 38 


48 


C 


Bayes 
L.S. 


.31 

.31 


. 16 

.16 


.43 
.43 


.29 
.27 


.48 
.47 


.41 
.42 


.47 
.46 


116 


D 


Bayes 
L.S. 


.15 
.15 


.08 
.08 


. 33 
. 33 


.14 

. 12 


.29 
.32 


. 31 
. 30 


. 30 
.30 


204 


E 


Bayes 
L.S. 


.40 
.40 


.43 

.43 


.54 
.54 


.46 
.46 


.56 
.56 


.56 
.56 


.57 
.56 


151 


F 


Bayes 
L.S. 


.20 
.20 


.14 
. 14 


.48 
.A9 


. 19 
.20 


.48 

.50 


.48 
.48 


.50 
.49 


109 


G 


Bayes 
L.S. 


.34 


. 30 
. 30 


.58 
. 58 


.36 
.36 


.50 
.60 


.58 
.58 


.60 


93 


H 


Bayes 
L.S. 






WOMEN 


ONLY 










r 


Bayes 
L.S. 


.33 

. 33 


. 33 

-.32'^ 


.52 
.53 


.37 
-.37^ 


.56 
. 56 


.46 
. 55 


.56 
.56 


129 


.1 


Bayes 
L.S. 


.41 
.41 


. 31 
. 31 


.44 
.44 


. 38 
.40 


.48 
.49 


.48 
. A7 


.50 
.49 


112 


K 


Bay t_'S 
L.S. 






WOMEN 


ONLY 










L 


liayes 
L.S. 


. 36 
.36 


. 32 
. 32 


■ .55 
.55 


. 39 
. 39 


. 56 
.56 


.56 

.55. 


. 56 
.56 


202 



^COR is the correlation of predicted scores with observed scores. 



..S. stands lor It^asl squares. 



^Mlll tiplicaL i ^/e parameter was ne^;atjv^' 



'I 



Tabl*,' 



Values of COR i.n the Cros^s Samiil.e: Kc^malos 











Predictor Combination 




College 


System Type 


V 


M 


H 


r 




M,H 


V M H 


A 


Bay es 


.43 


. 34 


.68 


.46: 


.68 


.66 


.09 




L.S. 


.43 


. 34 


.68 


.46 


. 70 


.69 


. 70 


B 


Bayes 


. 49 


. 28 


.50 


.49 


. 55 


.51 


.55 




L.S. 


. 50 


. 28 


.50 


.-A 9 


.58 


. 52 


. 58 


' C 


Baye s 


. 54 


. ^7 


.43 


.58 


.62 


.40 


.59 




L.S. - 


.54 


.47 


.43 


.58 


, 59 


. 51 


. 60 


D 


Bayes 


. 24 


. 27 


.44 


.29 


. 30 


.42 


.43 




L.S, 


>.24^ 


-.27^ 


.44 


-.29"^ 


-.43 


.45 


.43 


E 


Bayes 


. 53 


. 50 


.66 


. 58 


.70 


.67 


.70 




L.S. 


.53 


. 50 


.66 


58 


.70 


.67 


. 70 


F 


Bayes 


. 2 7 


. 37 


.48 


. 36 


. 50 


.45 


.47 




. ■ L.S. 


.27 


. 37 


.48 


. 36 


.49 


.50 


, 50 


G 


Bayes' 


.57 


.44 


.62 


.58 


.73 


.64 


.72 




L.S. 


. 57 


.44 


.62 


. ''^ '^ 


.73 


.65 


.72 


H 


Bayes 


.40 


.43 


.60 


.47 


.62 


.63 


.63 




L.S. 


.40 


.43 


.60 


.47 


.62 


.63 


,63 


I 


Bayes 


.56 


. 52 


,72 


.60 


.76 


.75 


.77 




L.S. 


.56 


.52 


. 72 


.60 


.76 


.75 


.77 


J 


Bayes 


.55 


.52 


.66 


.60 


.70 


.66 


.71 




L.S. 


.. .35 


. 52 


.64 


.60 


. 71 


.67 


.71 


K 


Bayes 


.40 


.40 


.39 ■ 


.46 


.47 


.46 


.50 




L.S. 


.4u 


.40 


.39 


.46 


.47 


.47 


.49 


L 


Bayes 


. 5 2 


.46 


.65 


. 56 


.68 


.66 


.68 . 




L.S. 


,52 


.46 


.65 


. 56 


.68 


.66 


.68 



S ample Size 
199 

93 

120 

220 



116 



142 



138 



344 



15! 



128 



74 



353 



b- 



^COR is the correlation of predicted scores vith observed scores. 
L.S.. stands for least squares. 



"Multiplicative paraTnc^tor was negative. 



Table 6 



Values of COR in the Cross Sample: Combined Males and Females 



Predictor Combination 



Co] leet 


System Ty^^e 


V 


M 


H 


V M 

V ,11 


V H 

V , IL 


M U 


V M 


A 


Bayes 


.AO 


.30 


.66 


.42 


.70 


.66 


.70 




L.S.^ 


. 40 


30^ 


. 66 


.42 


. 70 


. 67 


. 71 


B . 


Bayes 


.49 


.26 


.41 


.49 


.52 


.45 


.50 




L . S . 


. 49 


. 26 


.41 


. 49 


. 50 


.43 


SO 


C 


Bayes 


.40 


.25 


.46 


.41 


.51 


.42 


.53 




L. S . 


. 40 


-.25^ 




41 


. 54 


. 47 


S A 

• J H 


D 


Bayes 


. 22 


.15 


.41 


.22 


, . 37 


.38 


.39 










A 1 


9 9 
. Z Z 


. 39 


. 40 




E 


Bayes 


.47 


,40 


.61 


.49 


.62 


.59 


.63 




Li • r) • 


A 7 




. u ± 


A Q 




69 




F 


Bayes 


. 23 


. 23 


.48 


.26 


.49 


.48 


.49 




T Q 


9 7 
. Z J 


. Z J 


A ft 

. HO 


9 c; 
. Z3 




Aft 




G 


Bayes 


.46 


. 31 


.6j 




.63 


.64 


.69 




J > • J3 • 


.HO 




. 0 J 


A A 


69 


. H 


Aft 
. DO 


H 


Bayes 


.40 


.43 


.60 


.45 


.62 


.64 


.63 




L. S. 


.40 


.43 


.60 


.44 


.62 


.62 


.62 


' 1 


Bayes 


.45 


.40 


.65 


.48 


.68 


.67 


.68 




L.S. 


.45 


.40 


.65 


.47 


.68 


.67 


.69 


J 


Baye.s 


. 50 


.41 


.60 


.51 


.66 


. 


.65 




I..S. 


. 50 


.41 


.60 


.52 


.66 


.62 


. fi6 


K 


Bayes 


.40 


.40 


. 39 


.45 


.48 


.49 


.50 




L.S. 


.40 


. 40 


. 39 


.44 


.47 


.44 


. 48 


L 


Bayes 


,44 


.36 


.63 


.46 


.57 


.60 


.65 




L. S. 


. 44 


.36 


.63 


.46 


.65 


.64 


.65 



S ample Size 
378 

141 

236 
424 



251 



231 



344 



284 



240 



74 



555 



lOR is tlu» correlation of predicted scores with observed scores. 
L.S. stands for least scjuares. 



Mui 



;>I Lcative parameter was negativt 



-38- 



Table 7 



Values of AR^ in the Cross Sample: Males 



Pred ic tor Comb ination 



School 


System Type 


V 


M 


11 


V,M 


V,H 


M,H 


V,M, 


A 


Bay eg 


- . 2.0 


^.06 


- . 20 


-.16 


-.27 


-.19 


-.20 


\ 


L.S. 


-.38 


-.19 


-.27 


-.23 


-.30 


-.25 


-.28 


B 


Bayes 


.34 


.31 


. 27 


.34 


. 32 


.31 


.29 




L.S. 


.38 


. 37^ 


.37 


. 37 


. 37 


.37 


.37 


c 


Bay es 


.02 


.06 


.09 


- .02 


- . 02 


.06 


.01 




L.S. 


-.06 


.06 


. 11 


-.02 


.07 


.09 


.06 


D 


Bayes 


.'23 


. 31 


. 16 


. 27 


. 18 


.19 


. 21 




L.S. 


. IB 


. 15 


. 12 


. 21 


. 15 


.16 


.16 


E 


Bayes 


-. 12 


-.05 


-.09 


-.06 


-.08 


-.06 


-.07 




L.S. 


-.13 


-.01 


-.12 


-.05 


-.11 


-.09 


-.10 


F 


Bayes 


-.35 


'-'.34 


-.47 


-.34 


-.47 


-.45 


-.46 




L.fi.. 


-.32 


-.28 


-.47 


-.27 


-.45 


-.44 


-.44 


G 


Bayes 


-.02 


.08 


-.02 


.04 


-.02 


.00 


.00 




L.S. 


-.06'= 


-.02 


-.05 


-.05 


-.06 


-.04 


-.05 



Bayes 
L.S. 



WOMEN ONLY 



I 


Bayes 


.06 


. 11 


-.01 


.09 - 


-.06 


.02 


-.01 




L.S. 


.16 


. 17 


^ .04 


.17^ 


.02 


.07 


.04 


J 


Bayes 


.03 


.07 


.21 


.14 


.27 


.31 


.29 




L.S. 


.09 


. 20 


.24 


.21 


.28 


.28 


.30 


K 


Bayes 
L.S. 






WOMEN 


ONLY 








L 


Bayes 


. 10 


. 20 


.14 


.16 


.14 


.19 


. 15 




L.S. 


.08 


.18 


.21 


.14 


.18 


.24 


.20 



^AR is the average difference between predicted and observed grades 

^L.S. stands for least squares. 

^'Mul tipl lea t Lve parameter was negative. 



Table 8 



Values of AR in the Cross Sample: Females 



Predictor Combination 



bcnoo 1 


System Type 


V 


M 


H 


V.,M 






V,M,H 


A 


Bay es 


.13 


-.03 


-.02 


.08 


.02 


-.11 


-.03 




L. S • 


. 11 


.03 


-.07 


.05 


-.01 


-.10 


-.02 


B 


Bayes 


.11 


.07 


.07 


.15 


.15 


. 11 


.14 




L, S • 


. 12 


. 15 


. 15 


. 15 


.20 


• 1^ 


. 20 


C 


Bayes 


.21 


.19 


.23 


.20 


.29 


-.26 


.27 




L. S . 


. 30 


. 22 


. 28 


.29 


.33 


.28 


.33 


D 


Bayes 


.28 


.35 


.21 


.27 


.22 


.23 


.20 




L, S . 


. 25 


. 26 


. 17 


.26 


.20 


.20 


.20 


V 


Raves 


.04 


.08 


.14 


,06 


.11 


.11 


. 10 




L. S. 


.05 


. 11 


. 13 


.07 


.11 


.14 


.11 


F 


Bayes 


-.30 


-.29 


-.31 


-.28 


-.28 


-.28 


-.27 




L. S . 


- . 16 


.22 


. 28 


-. 17 


-.23 


-.25 


-.22 


G 


Bayes 


.24 


.27 


.16 


.24 




.15 


.16 




L. S . 


. 21 


. 22 


. 10 


.2] 




.13 


.14 


H 


Bayes 


-.07 


-.01 


-.05 




-.08 


-.06 


-.10 




L.S. 


-.05 


-.04 


-.06 




-.09 


-.06- 


-.09 


T 


Bayes 


.09 


.09 


.Ob 


.08 


.00 


-.02 


-.01 




L.S. 


.OA 


.07 


.02 


.04 


-.05 


-.01 


-.05 


J 


Bayes 


.20 


.19 


.19 


.23 


.21 


.21 


.22 




L.S. 


.26 


.30 


.22 


.27 


.21 


.2A 


.22 


K 


Bayes 


. 16 


. 15 


.08 


. 10 


-.13 


■^.A6 


-.32 




L.S. 


10 


-.36 


-.15 


-.35 


-.27 


-.39 


-.34 


L 


Bayes 


.20 


.25 


.08 


.14 


.09 


.11 


.09 




L.S. 


.17 


.19 


.10 


f 


.10 


.11 


.10 



Al^ is the average difference between predicted and observed grade 

h 

L.S. stands for least squares. 
^Multiplicative parameter was negative. ^ 
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Table 9 

Values of AR^ in the Cross Samples: Combined Males and Females 



Predic tor Comb iiiation 



School 


System Type 


V 




H 


V ,M 


\T IS 


M , n 


\T Tuf U 


A 


Bayes 


.08 


.05 


.03 


.07 


.OA 


.04 


.OA 




L. S . 




- . Ul 




. U J 


. OA 


. 02 


. U J 


B 


Bayes 


.04 


.01 


-.02 


.06 


.OA 


.00 


.03 




L . S . 






. U J 


. U J 


.05 


.03 


. U J 


C. 


Bayes 


.06 


.12 


.15 


.07 


.13 


.17 


.13 




L. S . 


. Uo 


. IZ 


. 1 / 


. lU 


. 16 


. 18 


1 c 

. lo 


D 


Bayes 


.26. 


.25 


.20 


.24 


.17 


.17 


.18 




L. S . 


. 20 


. 19 


. 17 


. 19 


. 17 


. 17 


. 17 


E 


Bayes 


-.04 


.OA 


.01 


-.02 


-.05 


.00 


-,02 ■ 


• 


L. S. 


- . 04 


. 09 


.01" 


- . 01 


- . 01 


.02 


. 00 


F 


Bayes 


-.53 


-.50 


~.5A 


-.54 


-.55 


-.56 


-.55 




L. S . 


- . 47 


- . 52 


-. 57 


- . 4 / 


-.53 


-. 57 


- . 53 


G 


Bayes 


.24 


. 29 


.25 


.26 


.25 


.24 


.25 




L. S. 


. 32. 


. 3A 


. 27 


. 32 


. 29 


. 28 


. 29 


H 


Bayes 


"-'.13 . 




- .16 


-.lA 


-.16 


-.17 


-.18 




L.S. 


-.22 


- . 22 


-.22 


-.22 


-.23 


-.23 


-.23 


I' 


Bayes 


-..01 


-.05 


.00 


.00 


.OA 


-.02 


.02 




L.S. 


.02 


-.00 


.01 


.03 • 


.05 


.02 


.05 


J 


Bayes 


.18 


.16 


.30 


. 21 


.37 


.33 


. 31 




L.S. 


.29 


. 23 


.39 


. 30 


.AO 


.40 


.AO 


K 


Bayes 


.11 


.15 


-.09 


.09 


-.10 


-.15 


-.23 




L.S. 


-.07 


-.07 


-.09 


-.12 


-.28 


-.18 


-.29 


L ' 


Bayes 


.19 


. 24 


.19 


.19 


. . 21 


,21 


.18 




L-.S.. 


.20 


. 2A 


.20 


.21 


.18 


.«21 


.19 



AR is the average difference between predicted and observed grades. 
^L. S . stands for least squares. 
Multiplicative parameter was negative- 
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Table 10 

^ Values of VR in the Cross Sample: Males 

Predictor Combination 



College 


System Type 


V 


M 


H 


V,M 


V,H 


M,H 


V,M,1 


A 


Bay es 
L.S.^ 


. 36 


. 33 


.24 


. JU 

.30 


. 21 


.23 


. ZD 

.21 


D 
O 


tsayt^o 
L.S. 


. Jo 
.37 


/i /i 
.45" 


.42 


. jy 

.42 


. Jo 
.40 


A'^ ' 

.42 


. Jy 

.41 


c 


Ddy fcib 

L.S. 


.51 


. ju 
.51 


A 1 

.41 


A A 

.51 


. JO 

. 39 


. ^ J 

.43 


• J-7 

.40 


D 


Bayes 
L.S. 


. 27 
.27 


. 30 
.22 


. 25 
.23 


. 31 
.26 


.-31 
. 27 


126. 
.23 


.29 
.27 




Bayes 
L.S. 


. Jo 
. 38 


. J / 

.38 


. J J 

.33 


. Jo 
. 17 


. j2 
. 32 


o o 
. JZ 

.32 


. JZ 

.32 


r 


Bay es 
L.S. 


.63 


fill 
. .66 


SI 

.50 


.66 


s 1 
.49 


sn 
.50 


50 
.49 


G 


Bayes 
L.S. 


. 30 
.34^ 


. 31 
. 32 


.24 
.30 


. 30 
. 32 


. 27 
• 31 


.25 
.29 


.23 
.30 


H 


Bayes 
L.S. 






WOMEN 


ONLY 








1 


Bayes 
L.S. 


.63 
.68 


.65 
.69^ 


.53 
.56 


.63 
.70^^ 


.50 
. 53 


.58 
.56 


.51 
.53 


J 


Bayes 
L.S. 


.40 
.40 


.43 
.46 


.40 
.40 


.41 
.41 ■ 


.38 
.38 


.39 
. 39 


.37 ^ 
. 38 


K 


Bayes 
L.S. 






WOMEN 


ONLY 








L 


Bayes 
L.S. 


. 27 
.27 


. 28 
. 28 


.22 
■. 26 


.27 
.26 


. 22 
.26 


. 22 
.26 


■ . 22 
.26 



is the variance of the residuals. 

^L.S. stands for Least squares, 
c 

Multiplicative param'^^ter was negative. 



Table 11 

Values of VK'^ in the Cross Sample: Fe:aales 
Predictor Combination 



College System Type V M U M,H V,M,H 



A 


Bayes 
L.S. 


.23 
.29 


. 31 
.32 


.19 
.18 


. 27 
. 28 


.18 
.18 


.19 
.19 


. 18 
. 17 


B 


Bayes 
L.S. 


.26 
.27 


. 30 
. 30 


.25 

.26 


. 26 
.26 


.24 
.22 


.25 
.25 


.23 
. 22 


C 


Bayes 
L.S. 


.21 

.22 


. 23 
. 25 


. 25 
.24 


.20 » 

.22 


.19 
.20 


.25 
. 22 


. 20 
.18 


D 


Bayes 
L.S. 


.18 
.18 


.18 

c 

. 19 


.16 
.17 


. 18 

.18^' 


.20 
.15 


. 17 
.15 


.17 
.15 


E 


Bayes 
L.S. 


.36 
. 36 


.37 
. 37 


. 28 
.29 


. 34 
.34 


.26 
.27 


.27 
. 28 


.26 
. 26 


F 


Bayes 
L.S. 


. 36 
.37 


. 34 
. 34 


.30 
. 30 


. 34 
. 34 


.29 
.29 


.31 
.29 


,30 
. 29 


G 


Bayes 
L.S. 


. 20 
.22 


. 23 
.23 


. 20 


1 9 
.20 


. 14 
.14 


. X / 

.18 


.14 


H 


Bayes 
L.S. 


.42 
.47 


.40 
.41 


. 31 
.38 


. 39 
.43 


.33 

.35 


.31 
. 35 


.30 
.34 


I 


. Bayes 
L.S. 


.40 
.38 


.43 
.41 


.28 
.30 


. 38 
. 36 


.24 
.28 


.25 
.29 


.23 
. 27 


J 


Bayes 
L.S. 


.44 
.47 


.48 
.50 


.36 
.36 


.42 
.44 


.31 
.30 


. 35 
. 34 


.31 
. 30 




Bayes 
L.S. 


. 22 
.27 


.23 
.44 


.23 
.25 


. 21 
. 31 


.22 
.23 


.37 
. 26 


. 25 
. 23 


L 


Bayes 
L.S. 


. 20 
.21 


.21 
. 22 


.16 
.17 


. 19 
. 20 


.15 
.16 


.16 
.16 


. 15 
.16 



is the variance of the residuals. 
^L.S. stands for least squares. 
Multiplicative parameter was negative. 



Table 12 



Values of VR*^ in ihv Cross Sample: Combined Males and Female 



Pred ictor Comb inat ion 



College System Type V M H VJl NLJi V,M, H 

A Bayes .30 .33, .22 .30 .20 .23 .20 

L.S. .31 .34^ .23 .31 .20 .22 .20 

_B Bayes .31 .35 .32 .31 .28 .30 .28 

L.S. .33 .36 .32 .32 .29 .31 .29 

C Bayes .34 .38 .32 .34 .30 .33 .29 

L.S. .37 .38^ .32 .36 .29 .32 .29 

D Bayes .23 .22 .19 .24 .25 .21 .22 

L.S. .27 .24 .19 .31 .23 .20 .23 

L Bayes .38 .42 .31 .37 .30 .33 .29 

L.S. .40 .41 .33 .37 .31 .33 .31 

K Bayes .48 .48 .39 .48 .38 .39 .39 

L.S. .51 .48 .42 .51 .40 .41 .40 

G Bayes .27 .31 .22 .27 .21 .21 .19 

L.S. .31 .31 .21 .24 .20 .20 .20 

II IJayes .42 .42 .32 .40 .-^0 .29 .30 

L.S. .42 .44 .31 .40 .30 .30 .30 

I 'i .Bay^s .54 . 5'8 .41 .52 .37 .38 .37 

' .53 .56 .41 .52 .37 .40 .37 

J Bayes .44 . 51 .38 .43 .34 .41 . 34 • 

L.S. .45 .56 .39 .43 .34 .37 .34 

K Bayes .22 .22 .22 .21 .20 .20 .19 

L.S. .22 .22 .22 .21 .20 .21 .20 

L Haves .24 .26 .18 .24 .21 .20 .18 

L.S. .25 .27 .10 .25 .19 .19 .19 



"^VR is the variance of tlio residuals. 

b , _ , 

L.S. stands tor U'ast sc|uart's. 

c 

Mu 1 1 i p 1 ira t i vt^ parameter was n('^;aLiv<'. 



Table IJ 

Values of ZOI/^ in the Cvohs Sample: Males 



Pr ed ic tor Comb Ina t ion 



School Syste m Type X il IL V , M V,H M^IJ V,M,H 

A Hayes .36 .42 .A5 AO .37 .45 .4^ 

L.S.^ . 32 .34 .40 .36 . i9 .41 .41 

B Bayes .35 .25 .23 .31 .31 .31 .25 

L.S. .33 .23 .25 .25 .27 .25- r27 

C Bayes .49 .39 .41 .49 .46 .41 .42 

T.S. .37 .39 .40 .45 .43 .39 .41 



Bayes .32 .31 .39 .32 .38 .39' .^'^0 

T.S. .35 .37 .40 .37 .43 .40 .41 

Bayes .42 .4 1 .42 .40 .40 .45 .43 

L.S. .41 .44 .39 .42 .42 .43 40 

Bayes'' .47 .48 .46 , ''6 , . j 

L.S. .45 .45 -4 ■ .4t . ^6 . 44 .46 



G 


Bayes 


. 37 


.33 


.41 


.33 


.37 


.39- 


.41 




L.S. 


.33 


.31 


.35 


.'33 


.35 


.37 


.35 


II 








WOMEN 


ONLY 








I 


Bayes 


.40 


.42 


.42 


.41 


.42 


.43 


.43 




L.S. 


.43 


.43' 


. .43 


.43 


.43 


.40 


.42 


J 


Bayes 


.46 


.46 


.46 


.46 


.43 


.44 


.40 




L.S. 


.39 


.42 


.44 




• .42 


.44 


.42 


K 








WOMEN 


ONLY 








L 


Bayes 


■ .44 


.40 


.44 


.38 


.45 


.39 


.40 




L.S. 


.43 


.41 


.38 


.41 


.42 


.33 


.39 



*V.OL is the average of a variable. 
^L.S. stands for least squares. 





Va 1 lies 


of ZOL 


in cliLi 


Or OSS 


Samp le : 


Females 
















Predictor Comb inatioa 







)chool 




V 


M 


11 








VJ^il 


A 


f ^ i I V c • s 


. 38 


. 4 2 


. 50 


.37 


.52 


. j1 


. 51 




i...s.t' 


.37 


.43 


.49 


.37 


.51 


.48 


.48 


B 


f^c'i V OS 


. 




. 43 


. 44 


.48 


. ^ / 


. 4 5 




L.S. 


• . 4 3 


.43 


.4 5 


.44 




47 


A 1 
.Mi. 


C 


1 u I V I'S 


. 33 


. -) / 


. 28 


. 37 


. 38 


. 11 


. 35 




I..S. 


. 30 


. 30 


.28 


.35 


.36 


. 30 


.36 


I) 


Ka ves 


. 30 


'> ^ 


. U) 


. 30 


.33 


. 33 


. 3 5 




f-.S . 


. 3 5 


.35 


.35 


.35 


.35 


.33 


.35 




av 


. 47 




. 50 


. St ) 


. 53 


. j4 


. 53 




1..S . 


. 4 7 


.47 


. 4 9 


.48 


. 54 


. 52 


. 55 
























. 4 2 


. 4 2 


.•40 


. 42 


. 39 


/. /. 


. M 2 




L . S . 


. . 4 f ) 


.46 


.42 


.46 


.44 


.42 


.42 


c 


1 J a V ^ s 


. 4 1 


. 33 


. 1 


. 4 3 


. 51 


A 1 


. JU 




1 . . S . 


.4 1 


. 34 


.44 


.43 


. 53 


.46 


. 51 


[I 


[ i ; J (^1 ^ 


. 4 1 


. 4 


. 51 


.42 


.47 




. 50 




1.. s . 


. 3 5 


.40 . 


.43 


. 38 


.46 


.46 


.47 


1 


Wil Y (J j.^ 


. i9 


. -t ^ 


. 52 


.40 


. . 58 


. 55 


. 58 




i . . s . 






. 54 


.43 


. 54 


. 57 


. 54 


J 


f)a yes 


.44 


.4 5 


.48 


.48 


. 55 


.48 


. 55 




L. S . 


. \ 2 


.44 


.48 


.44 


. 55 


.48 ■ 


.54 


K 


i;.ivt's 


. rj 


. 30 


.34 


. 34 


.46 


. 28 


.31 




I., s . 




. i5 


. iH 


.26 


.38 


. 34 


.36 


L 


ria vl'S 


. ;! 


. i9 


.47 


.44 


.48 


.48 


. 50 




i,.S . 




. ■ ♦ 1 


. 4 7 


. 4 2 


. 50 


.48 


.48 



y-^H. is rlic avc'r.'i,t;i' of .i v.ir i.ili 1 ^ , 
'l.S. stands for Least squaic.s. 



ERIC 



ERIC 



Table 1") 



Values of ZOL""^ in t:ho C^rcjss Sanple; (Mipihined Males and l-'emales 



Pred I<; tc^r ('onibinatiun 



^JrA9.9.1_ L^JLti^5LJ!ip.^' Y ''^ ^[ ^L^^^ ^J;^ M,H V^MjH 

A Bayes .39 .39 ,50 ,39 .52 .50 .52 

I..S.^' .^0 .37 .51 .AO .52 .A9 .52 

H Bayos .AO • .41 .38 .42 .41 .41 .4] 

L.S. .42 .41 .37 .41 .38 .38 .39 

C Bayes .42 .37 .36 .42 .39 .33 . AO 

L.S. .39 .32 .35 .42 .41 .36 .39 

n Bayes .32 .33 .39 .35 .36 .38 .38 

L.S. .35 .32 .39 .33 .38 .39 .39 

Bayes .4 5 .41 .48 .43 .51 .46 .50 

L.S. .43 .38 ./i6' .43 .49 .46 .49 

F Bayes .35 ' .32 .32 .35 .32 . 32 . 32 

. L.S. .4 1 .31 ,30 .42 .34 .31 .34 

C: Bayes .38 .32 .40 .37 .41 .40 .39 

L.S. .31 .32 .41 .33 .40 .41 .40 

H Bayes .42 .35 .48 .40 .51 .50 .52 

L.S. .38 .35 ./*8 .39 .50 .50 .50 

I Bayes .41 .40 .48 .42 .50 .48' .49 

L.S. .40 .39 .48 .42 .49 .49 .50 

J Bayes .48 .43 .46 .48 .41 .44 .47 

L.S. .41 .41 .42 .43 .40 .40 .41 

K Bayes .32 .35 .41 .34 .46 .41 .46 

L.S. .36 . 35 .41 . 39 .^3 .4:^ .42 

L Baves .39 .38 .44 .39 .42 .42 .46 

L.S. .40 .37 .43 .39 .45 .42 .A4 



a 



ZOL is the average of a variable. 
^ L . S . s tands for least squa re.s . 



Tablt- \h 



Values ot Af-:'^ in the Cimss Sample: Females 



Predi\;Lor Combination 



St-' ho u I System Typ e V M I! V V,H N,H y ^M, H 

A Hayes .44 .44 . .42 .35 .35 .34 

I-S.^ ,44 .44 .3^; '.43 .34 .35 .34 

is Hayes .39 .42 .40 .40 .41 .40 .40 

.AO ,43 .43 .40 .40 .41 .40 

C Hayes .42 '.43 .46 0 .42 " .48 .42 

I..S. .47 .46 .48 .45 .43 .46 .44 

I) Hayes .42 .47 ,37 .42 .40 .38 .37 

L.S. .40 .4] .36 .40 .36 .36 .36 

K Hayes ' .46 .48 .^3 ,45 .41 .42 .41 

I^-S. .46 .48 .44 .45 .42 .43 .42 

K Hayes .50 .49 .48 .48 .46 .46 .46 

I-S. .46 .46 .46 .44 ,44 .45 .44 

Hayes .40 .45 .37 ,39 .33 .35 .33 

I-.S. .40 .44 .37 .38 .32 .35 .32 

H Hayes .51 .51 .42 .49 .45 .44 .42 

l-.S, .56 .52 .49 .53 .47 .47 .47 

I Hayes .53 .53 .42 .51 ,37 .39 .37 

1^.^. .50 .53 .42 .49 .41 .41 .40 

J Helves ,55 .56 .48 .53 .'4 5 .48 ,45 

h.S. ,57 .60 .49 .56 .45 .48 .45 

K Hayes ,41 .42 .40 .39 .38 . 62 .49 



I^.S. .42 , 59 .40 . 54 ,41 .51 



\'>riycs .40 .42 .3 3 .38 . 32 .33 . 31 

I-.S. .39 .40 .34 ,38 .33 .33 , 33 



a . 



AE is the average of the absi^lute value cif the (llfrerence between the 
pri'Jirt. ed Ki'<ide aa<l obsrrvetl .^rade. 

b 

L.S. stands fur least squares. 



-►48- 



Table 17 

Values of AE'^ in the c:ross Sample: >Iales 



Pr edlc tor ('oinbinat ion 



O ^ 1. ^ 1 

School 


Sys t em Type 


V 


M 


H 




V,H 

.47 
.45 


M,H 

.41 
.44 


V,M,H 


A 


B.'ives 

r - h 

L. b . " 


.49 
. 58 


.46 
. 49 


.42 
. 46 


. 4 5 
. 48 


.40 
.45 


B 


Bay e.s 
L.S. 


. 59 
. 60 


.62 
. 65 


.60 
. 6 3 


.60 
.63 


.57 
.62 


.62 
.63 


.59 
.62 


C 


Bayes 
L.S. 


. 50 
. 55 


. 54 
. 55 


. 51 
. 51 


. 50 
. 53 


.47 
.48 


.51 
.51 


.48 
.49 


I) 


Bayes 
L.S. 


. 44 
.42 


. 48 
.40 


. 38 
. 37 


.46 
.42 


.38 


.39 
.38 


.40 
.38 


K 


Ikiyes 
L.S. 


. 50 
.51 


. 48 

.^9 


. 47 
.48 


.49 
.49 


.46 
.47 


.45 
.46 


.45 
.46 


F 


Btiyes 
L.S 


. 6 '3 
.6 ] 


. 6^+ 
.64 


. 63 
.64 


. 63 
.64 


.63 

.62 


.62 
.62 


.63 
.62 


G 


Bayes 
L.S. 


.45 
. 50 


.47 
.48 


.41 
.47 


.46 
.49 


.44 
. 48 


.42 
. 46 


.40 
. 47 


H 








WOMEN 


ONLY 








I 


Bayes 
L.S. 


.64 
.66 


.64 
.67 


.59 
.60 


.64 
.67 


.57 
.59 


.61 
.60 


.58 ■ 
.59 


J 


Bayes 
L.S. 


.48 
. 50 


. 5.1 
.54 


. 52 
. 53 


. 51 
.52 


.53 
.53 


.54 
.53 


.53 

.54 


K 








WOMEN 


ONLY 








L 


Bayes 
L.S. 


.42 
.42 


.46 
.45 


.39 
.44 


.43 
.43 


.40 
.43 


.42 
.47 


.40 
.44 



AF Ls tfie average of the absolute value of tiie difference between the 
predicted grade and observed grade. 

^L.S. stands for least squares. 



Table 18 



\'aliius r-1 Av: in Liu- ciruss Sample: Ciumbined Ma i tis and 1- enia U\s 



School Sys tern T^^^ 



Ba Vfs 

I.. S . 

I^a-VL's 
I. . S . 

f)a ves 
L. S . 

Hayes 
(. . S . 

Hayes 
L . S . 

Ba yes 
L.S. 

Hayes 
I..S. 



L.S ./ 

/ 



jMves 
' f.. S. 

ii. I ye s 
L.S. 

P,aves 



.44 
.45 



.45 

.45 
.48 

.43 
.43 

.49 
.49 

.66 

.6'3 

.47 
.52 



\ 



. 54 

: 59 
. 59 

. 5 3 
. 

, 40 
, 39 



. 4 '3 



M 

.46 
.50 

.47 
.48 

.50 
.57 

.43 
.42 

.52 
.53 

.64 

.65 

. 52 
.54 

.54 
.56 

.61 

.60 

. 58 
. 62 



i9 



46 



Predictior Combination 
H 



\ 



.37 
. 38 

.46 

.46 

.48 
.48 

. 37 

.36 

.44 

.46 

.64 
.69 

.43 
.43-- 

.44 
.45 

.51 
.57 

. 53 

. 57 

. 38 
. 38 

. 38 
.39 



.44 

.45 

.44 
.45 



.45 
.47 

.43 



.49 

.49' 

.66 
.63 

.47 
. 52 

.51 ] 

. 58 
. 58 

. 53 
. 56 

. 19 
.39 



4 3 



, .35 
.43 



,45 
.45 

.39 
. 38 

.43 
.44 

.65 

.65 

.42 
.44 

.43 
.44 

.48 
.49 

. 37 
. 56 

. 36 
.40 

.40 

. 38 



. 38 
.37 

.45 
.45 

.49 
.43 

.37 
.36 

.45 
.46 

.67 
.68 

.42 
.43 

.43 
.45 

.49 
. 50 

. 56 
.57 

,38 
,38 

39 
39 



V,M,H 

.36 

. 35 

.44 
.44 

.45 

.45 

.37 
.38 

.42 
.44 

.65 
.65 

.41 

.44 

.43 
.44 

.48 
.48 

. 52 
. 56 

.38 
.4] 

. 37 
. 38 



a 



\K Is tin 



le averar.e nf thr absolute v'aliie ef t lie ditTereiiee betv^eeii tbe 
predi.c;teci ^rade and n[^serve:i ^rado. 

L.S. stands for least squares. 



Soc i'ararvtr; 



AvoraRL' lA'asi 



i 



V^' .02 



II'' 



l.il 


.')'! 


i ,0..' 




.Oi'.2 


..0')2(. 








.03 


.02 


.()] 


.99 


.94 






■ .008? 




l.'j80« 


l.'llhH 




.71 


..(11 






.V) 






,2/2n 





.^^ Ml 
.(i(i;sii - nod? 



.0/^ ,i: .ss 



.020'j - .0087 .Oilif) JlOM .(Mi; .00?.'^ 

^ ^4).'f, .:Mn/ 



,1 .M) .Vi .1'' .-■> .VI 



.(^^ifi .fvv) .H/ui 



.02 


M 


A.! Jil 


.:i2 .(12 .01 




.HI) 


.27 .72 


.!■) .')') 1,!5 


.026'i 


.(il21 


.04ilh ^ .017') 


.mi .012'. .omo 


.11% 


I.MIC 


1 - .807« 1.259'. - 


.|7q^, - .'WVl 2.1505 



.05 


.11'. 


.01 


.02 


.02 




- .01 


.47 


.25 


.5') 


. 55 


.60 


,74 


.'IB 


.94 - 


.O'i 


.10, - 


• .1') - 


.24 


.21 


1.50' 


. 5809 


.2Hih 


.h:)i2 ■ 


.f)25^ 


.fi889 " 


. 787(1 


.0910 
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.31 


.23 


,27 


.32 


vr 


.If) 


.35 


.31 


.38 


.36 


.35 


,30 


.39 


QP 




.21 


.23 


.26 


.29 


.23 


.2^ 


,31 




/17 


.35 


.31 


M 


.35 


.3: 


.28 


,40 


pr 


.'Jo 


.36 


.31 


.39 


.37 


.36 


.34 


,41 


VQP 


,2^ 


.21 


.25 


.29 


.37 


,27 


.30 


,40 


VQU 


.37 


.36 


.31 


M 


.41 


.^0 


.32 


.47 


VPU 


.% 


.36 


.32 


.39 


.41 


.38 


.36 


.45 


QPU 


.37 


.37 


.31 


M 


.42 


.39 


.33 


.46 


vqpu 


.38 


.37 


.32 


M 


.46 


M 


.35 


.52 



System Vai;dity Coefficients for CEEB VSS Data in the Back San-.ple 

Ji^^l?X ^^^^i^^-?-^ Bayes 



Predictors M I ^ I 

V ,35 .50 .39 .27 .46 .35 

M .37 .43 .31 .30 .39 .23 

H .58 .56 .35 .60 .54 

VM .3^) .53 .^0 .39 ,48 .38 

VH .59 ,b'' .59 .59 .59 .60 

MH .58 .64 .57 .59 .67 .59 

V^I .59 -67 .59 .58 .70 .5^ 
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In liu- i)rK-stMU sludy, one liopes to produce numerical weigl^ts which can he 
used in r : -nb i m. : i on with nredi(^t-:^r scnr*--; i:^ (-onpncc- prediction:^ of )^rade 
point avera,^tvs ttint vould be achieved in t^raduare school. However, ic is 
known ahead of Lime that there will not he enough cases to dc a separate, 
stand-alone study at each school; pooling of data will be necessary, and this 
P<n)iLnp, will entail the use of some co.ivention to relate the weights used for 
differenl schools. In this Appendix, the weights used at the different 
schools will h.- f)roport ional , i.e., ihat tlic ratio of weights will be pre- 
served. In addiiion lo ihe proportional adjustment of weights at each school, 
a shift of means will also be incorporated. The hope of this appendix as well 
as ot Appendix .H is that, except for differences in difficulty and reliability, 
Lhe grades measure ihe sa^ne thing. 

For estimation piirpo.-.t-.; , it is assumed that data are available for 
samples of studorus t rom each o, a numbor of institutions. The weights to be 
used would be r.\\oscn so as to minimize the sums of squares of errors of estima- 
tion of t:he observed i-.radr point averages by the weighted sums of predictors 
scores. Th.cU sum sciuares is written as follows: 



■ . ( V 
1 i i 



(1) 



wise re 



s a snhsoripr indicating, the school; 

s a subscript indicating student within school; 

s a subscript indicating: tlu^ predictor variable; 

s an arbitrarv weigjit which was taken as \ini ty in the 

resru t s t udv ; 
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Y Is the p,rade point avera^^e of the j t_h student at the Ith 
i.i ' ^ 

school ; 

B allows for a shift of means; 
i 

b is the weight used for the Rth variable at the ith school; 
ig 

X is the score achieved on the ^th predictor by the j_th person 
at the i th school. 

It f-s a well-known result in least squares analysis that the value of 

that minimizes ^ is the mean of the rest of the values in the pare r.ch'.:>es 

of (1). That is, 

B. = • :: b. x. x (2) 



where the bar-dot notation is the familiar one indicating the averaging 
process. If the right hand side of (2) is substituted into (1)^ the effect 
is to replace the observed predictor and criterion scores by their deviations 
from school means. Then (1) ^Hn be rewritten 

't=Zw.(y..~Zb. x..)^-f2EA. (b. -a. 6) 
ij ^ g ig ^ S 



0 (P - E B 3 , r ,) . ■ (3) 



including all of the desired constraints. The quantities X are La.^^range 
multipliers included to incorporate the constraints that the weights will 
be proportional. Note that the constraint includes a product of the value 
a.^ which is the constant of proportionality for school i , and the weighj: 
3 for the predu'tor variable. Then the a's could all. be multiplied by 
some number and tlie i^- ' s d Lv ided by t'no same number and equat i on ( 3) v;ould 



remain essoiitial.ly unaffjected. Tlip c}u>ir.»> c^f t.ho scale of the a's and l) Vs' 
is immaterial, |/ut f or^ r.umo r ieal purposes one' must ho chosen. This choice is^ 
made accordinp, to the relationship in parentheses mu 1 1; ip] ied. hy 0 in equa- 
tion (3). 6 . is also a Lagrange mul t LpLler used to enforce nlie constraint. 
P i'o positive, and the' rVs form a matrix of full rank. 

To ob.tai.n (4), (5); (6), (7), and (8), rearrange the result of differenti- 
arin^:^ f V) wi th respect to h. \ a. , B \\ f^nd G respectively. 



(v. . - >: h. v_ ) X. . , \. 0 . C4) 

I'll 1(711(7 ; w> ' — f> ' ^ ' 

J 



ig « 



(5) 



. '1. ' : , r , - 0 . (6) 



h. = .1. .- • . ^ . (7) 

ig L g . ^ ^ 



(8) 



"Mm 1 r ii^lying (6) by ^ , summing over - g ■, and using (5) and (8) it can 
be si^en i:h»at • equals zero. Then jnul t ip lying (4) by \i , , summing 
ovvf , aiKl using (5) and (7) yieiiis 

.1. = [■ y. , (■ ■ ■■„, , ,>'!■/ [:: ;■: v. , (:: x. . x. . ,) ] (9) 

Mu 1 t 1 p Iv i n>', f/i) hv a. ^, summing, over i , using (6) and (7), and • 
ri'iiii'mber i Hi'/" tfiaL equals /.L^.r^, obtain ^ 

g . ' 1 ■ ^J«- : 11, 111]!'/ ■ • 
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^(luaL" ions (<S), (9), .uul (10) provich' Lht' i t ra t i nt'S by whii:li Ji solution i,s 
found. In thr- program which was used in t.lu' present st:udv r , was taken 
a s un i. t y if g= ^ ' , /. e r o o t: 1 1 e n^^ L s e , and V was t ake n as unity. A s t a r t i p.g 
point for the iteratious is to set all the . 's equal to unity divided 
bv I tie square root of the number of predit:tors. Then (9) can he used to 
find the a's , the resu I L i nr. a's can be used in (10) to .f^et ' s 
whieb can be normed to satisfv (8) and the results substituted in (9) 
and so on. Wiien the • ' s and a's cease to cb.angc from iteration to 
iteration, equations ("7) and (2) may be used to recover the b's and s 
One can develop an analog to the multiple correlation coefficient 

in that p is a sum of squares of residuals that can be subjected to a 
percentage comparison v;ith the S'am of squares around institutional means. 

A svstt.'n) coefficient th.'it has thie desired property is 

R - A - (:/ w,y^) . (ii) 

V i.j 

The denominator of the' quantity in parentheses in (11) is the sum of squares 
of resi<:k>als after fit i uR the institutional means; the numerator of that 
quantity is the sum of squares of residuals after fittini^ the institutional 
means and the predictors. Unity minus that fraction can be interpreted as 
the percent of varianct^ attributable to the pred i c t ors , and the square 
rootin,K completes the analogy to the multiple correlation coefficient. 

On oc CIS ion, in some sets of data where a school is represented 
by very few cases, will occasionally find a multiplicative constant, 
a , to lu^ ne^^ative while the others are positive. This mav be due to 
a reversal oi' lIic i^'ad*.- s^'aK' fi'om that school oi- f. o some real 

d i f f (M-v'iH^tj b(.'tween th/ii sehool and t'he hers in the analysis. Mc^st. 



likely, It t\\v (M-icurioM srale is not rwvr^c.d ruiaUve to the others, the 
reversal in ^[i\n is aot. lo be believeti, in the opinion of the auth.or. If a 
prediction on the ^u'ade point scale is to be recommended for that school, 
one should use the same si.^n displayed by the rest of the schools as a sign 
tor the mult i pi i c£it i ve constant for tlie school whose sign differs, then 
Lusinv^ (2) to adjust. li for that school. However, with so little data an 
attempt to put the predictions on t'ne scale for the particular school would 
probably await the accumulation of more data, using only the relative 
weLt;hts, the ;^ ' s , to eet predictions which are in good order hut not on 
the grade point S(\ale. 
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APPENDIX B 



As in the method of Appendix A, the present method accomplishes a 
pooling across institutions and does so by linear adjustments which 
differ at each institution. However, the adjustment is applied to the . 
criterion score rather than to the prediction. It is assumed that the 
samples available are the same as those in Appendix A, and that the s>jnbols 
i , j , g , Y. . , w. , and X.. all have the same interpretation. 
The sum of squares to be minimized, however, is 

'I- = T w. (A.Y. . - M. - Z h X )^ , (1) 

ij ^ ^ ^ ^ ^^''^ 

where the T-r s allow for adjustment of the means, and the A's adjust 

the criterion scores. The symbol h^ stands for the regression weights 

and only requires a single subscript since the step of partitioning the 

weight into regression coefficients and constants of. proportionality is already, 

in a sense, accomplished. As in the method of Appendix A, the value of 

is equal to the mean of the rest of the values in the parentheses of 
(1). That is, 

M. = A Y. - :: h X. . (2) 

If the right-hand side of (2) is substituted into (1), the effect is to 
replace the observed predictor and criterion scores by their deviations 
from school means. Then (2) can be rewr 1 1 1 en 
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where y is a [.arrange multiplior which imposes a scale constraint on 
the criterion scale. The constraint is needed because, as examination 
of the squared quantity in (3) shows, a trivial minimum of i/; can be 
obtained by defining all parameters equal to zero. The Lagrange constraint 
ensures that such a solution will not be obtained. 

After differentiating the following normal equations may be obtained: 



i 



A.- .c - .c^ H - Y .c = 0 ' , rA\ 

1 1 yy 1 xy i yy ' 



I ^ i'xy - I \ i^'xx H = 0 , (5) 



W. A. C 

1 1 1 yy 



The subscript C's are defined as follows; 



(6) 



c 

I XX 



X. . X 



1 



• - y 
J 



Multiplying (equation (4) bv 
V 1 o 1 ( i s 



summinr; over i , and using equation (6) 



> = 0) - •■ w. .C ' 11) (:: w C )' 
I L xy , 1 1. vv 



(7) 
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which may be substituted into (4) and the result solved for to 
obtain 



A, = c: ^ cr H + (Q - ): w, ll) {Y. v; .C ) ^ . (8) 

i i yy i xy ^ ^ i i xy ^ i i yy' 



The expressions for from (8) may be substicutcd into (5) to obtain 



fS w C - V C C ^ .C^ (J: w. C )(Z w c )~-^ 

^ i i XX ^ i i xy i yy 1 xy ^ i i xy _^ i i yy' 

{T. w C )'] . H = T. X . (9) 

, i i xy 5. w C i i XV 

i ^ i i yy i 



T!ie equations (9) can then be solved for H . 

The advantage of this method comes when variables are to be added 

as in a test selection scheme. The matrix to the left of (9) mu£ j be 

inverted in finding the vector H , but one could choose a method which 

could be worked a line at a time, such as the square root method. As 

Variables are added successively to a problem, the repeated development 

of the matrix to be inverted is unnecessary, whereas the method in Appendix A 

requires storage o: all the covariance matrices for each institution. 

Also, it can be shown merely by substituting in (3) for the A's and h's 

that fij = yQ > and since ij> is a sum of squares of residuals, /l-y is the 

analog to a multiple co7:relation coefficient. Reference to (7) shows 

that the computation of y does not require acc to the covariance 

matrices by institution. Furthermore, the formula for the covariance C 

' ze 

of an outside variable z , with the residuals e , after predicting y 
with the varj-ihlos x is: 



v.. 
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- [:: w. .c .c"^ .cr - (:: w. c )(t. w. .c )'"^(v w ) 

''^ I 1 1 1 yy L xy ^ i yz ^ x i yy \^ 1 i xy 

- X w. .cr ] H - Q [}: .C ] [Z w C l""^ . (10} 
^ 1 I xz ^ i 1 yz^ i i yy' 

If all passible predictor variables x included the variables x and z 

in' (10) , the distinction being that x had been used in previous calculations 

as predictors and z had not, then the quantities needed to compute C^^ 

would appear Ln matrices needed for (9) if all possible predictors were 

incJ.uded . One needs also to keep a column whose entries are E w, .C 

. 1 X xy 

-1/2 

Then at each stage compute K = C / ( Z w . . C ) for all z 

Z Z ^ 1 1 zz 

and ch(3ose' as the next predictor to be selected the one that yields the 

/ 

largest value of K . More complex variables selection schemes could be 
formulated, but this one seems simple and is analogous to the familiar 
Wher ry-Dooli t t le . 
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APPENDIX C 



To: Dr. It. F . Bolat 

Senior ii'e^-^t-arch ^.^ychologist 

Princetor. , New Jersey 



GRi: VALIDITY 3TLUY vUEGTIOIJNAJJU': 



Institution: 
Department : 

Name and Title of Person to Be Contacted; 



Name and Title of Person Compleiiiig This Form: 



I. Approximately how many applicants have you had, on the average per year, 
for the past three years? 



II. A. . Do you routinely compute a graduate grade point average? Yes No 

B. If answer to A is No, would you be willing to make copies of transcripts 
available tu GHF. Board foi' such com.putations? Yes No 

C. Please indicate the number of quality points used in your grading system. 
For exajT:ple, a four point system without D's and E's and in which wlLh- 
drawals (W's) are simply rot counted might look as follows: 

A y b 3 C ^ D ^W E ^f^U^? —/ Other ^j^^^ Other 

Please enoei' below the facts describing your system. 

A n n D E F Other 



vjt ncr 



III. P.c>v many cegree:^ grani'ed by your department last year? 

TY PK CP PFG:C :K NUl^ER GRANri-P-- 

} h . l) . 

V..^. or :!.A. 

Othci'-'" 



■ p r "in* 



r.-j-^r of saucer. :5 v;ho have successfully completed ihe fi.nr.l oriil 
for f-acP "Lypc .iv^^irr^e, daring the period Scpier-ber 1. 19*:^' th^:)^..^■; 
■• ^ , 1969. The Lr:o:v.C'd criterion for inclusion her'e is ichethr^r .klI ^"a-:pr^. 



:-.(a:e ;:dirar.e t.:>^ xLzlr- of grad:.a:e degree granted by your inst i:.ut 1 o:. 



V. 

o 

ERIC 
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IV. A. Are recor^ds for students who app;iied for adinission duiurig the pei'iod 

1965-00 and i966«67 available now (including those rejected)? 

Yes No 

B. If ansv;er to A is No, what is the earliest date for which complete 
records including rejectees would be available? , 

V. For purpo.'ies of the study, certain information will be needed for each student 
who vas granted a degree or who successfully completed a final oral examination, 
during the period September 1, 1968 through August 31 y 1969 and for those who 
applied foi* admission in 1965-66 and 1966-67 (or during the two yeai's following, 
the daLe given in question IV B, or until the present). The first group will 
be called "^^i-aduates" and the second group will be called ^'applicants . The 
final form In wnich data will be sought on graduates and applicants will be 
highly dependtjiit on the responses given by the institutions contacted. It is 
Lherefoi-.i uj*r;^;d that you ..ndicate below whetiier you can supply each type of 
iniomat.i'jn r-equt ?i','d . Please do not hesitate to include extended comments on 
the COMi irni/VriuM v/.HiiiKTo which are at the back of chis que stionnaire. 



Can supply data Can supply data Cannot supply 
for applicant tor graduates m data 



D 



□ 



□ 
□ 

□ 
D 
U 



D 



ERIC 



n 



D 
D 

! I 

□ 

n 



D 
□ 
□ 



□ 



u 



li, Gitri V Gcorc 
GRi Score 



^ccivea [ 



iione receive' 



flone received 



lJ 

n 



F. Ai'L-a of advancea test [if. other than econoraics)^ 

G. Advar.C'^d area teat scors 



None received | | 



Jndergra.duate grade poin^ average on a 4 point scale (GPn), or 
rank in class (and nuir.ber in class), or percentile rank in class 



I, Undergraduate Institution 



^' Date of first application of candidate for graduate work at 
ycv.r institution ^ , 



K. Terra to which the admission decision applied 



L, Nature of the first admissions decision Accept keject 

K. ^ Date of first enrollment as a graduate student at your 

institution ^ 



r 






N. Date of last enroUjnent 


















^1 Date of award (or expected ;award) of degree 



G 



?. Ty^jje of degree awarded (to be awarded) 

Q. If \iie -;-;(^ree is other than a Ph.D., is graduate work toward a 
d')c';orate suggested? (in plainer words, is he generally con- 
sidered Ph.D. ir^terial?) Yes Ko 
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CGNTINUATION SffiET 



Institut Li")ri : 
Department : 



ITZM Mirj^SER 



COMJ-ENT 



I 



ERIC 
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Graduate F^econd Examinations Board 



NoveiTioer i'^''/-' 



C.ha. "-nbf^ 
VtCh«et J f<rt'on»n 

Gtanitfy f roei 

Wayna C Han 
Nalionat R«8«erCh CounC:l 
N»;tOn«< A ;ac''imy * Scisnca« 

Joseph L McC»rihy 
Univrriity of We«hinglon 

Edward C Moure 
Mdssschuietts Board 
0* HtQ*>er Education 

J Boyd Pago 
CouMcii of Graduate School^ 

Michaet J Peic/ar. Jr 
'Jr»tv»r«ily of MerytOod 

(HiChard L Predrnora 
Ouke U'liversUy 

Mina Rees 
Yn* Cily University 
of Ne* ^orli 

S n Sh.riay Spragg 
ijn-vergiiy o* Bocheiner 

GeOrp'j ^ Springer 
University of Naw McsiCO 

Alien P Strehler 
Cern^gie- Mellon Univer«it 

Donald W Taylor 
^ate University 

Oarwin T Turner 
University of MiChigen 



The members of the Graduate Record Examinations Board, which 
formulates policies guiding the Graduate Record Examinations (GPIE), 
have been aware that research information supporting the use and 
interpretation of duta on which admissions decisions are based is 
indeed rparse. Suitable criteria for evaluating the outcomes of 
graduate education need to be developed, and the relations between 
these criteria and information available for admissions decisions 
must be disco*/ered. The Board is coirmittcd to mount systematic 
criterion develojm.ent and admissions research efforts. 

To these ends, scane steps ha\re recently oeen taken. . le Board 
and the National Science Foundation have jo.intly begun an empirical 
search of data fran Ph.D. 's in science areas to learn about special 
population effects on the relation between GRE scores and Lhe time 
needed to complete the doctorate. A study has been conducted re- 
lating the Test of English as a Foreign Language (TOEFL) and the 
GRE with foreign student success in graduate school. The develop- 
ment of a biograohical inventory to control the effects of motivat:on 
is being explored. The nature of the flow of graduating seniors to 
graduate and professional schools is being broadly surveyed under 
the auspices of the GRE Board, among others. Hov/ever, these and 
other research effort, s are complicated by the scarcity of clearly 
relevant evaluations of graduate school performance, and the linitod 
applicability of standard research and statistical techniques in the 
face of the very limited amount of data available. 

This letter has two purposes. The first is to inform you of 
some of the research efforts of the Board as described above. The 
second is to request your support in developing prediction techniques 
which are particularly applicable to graduate schools. Specia] new 
statistical techniques are needed in graduate admissions research as 
a consequence of the relatively small numbers of graduates produced 
at individual institutions (as compared with, say, recipients of 
baccalaureate degrees)^ These small number^ are further reduced at 
the der^rtmentai level. The techniques, being new, must be proven 
out on real data. Enclosed are some forms that will assist in the 
transfer of data from your institution to the Educational Testing 
Service who will conduct this project for the Boara. Two studies 
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Will bf: exx'cui.r^d, one in departments of psychology and the other in departments 
of economics. It ' our }\ope that these departments will supply necessary in- 
Tormation and participate in this study. 

We would appreciate your forwarding the enclosea materials to the chairmen 
ol the departments of economics and psychology at your institution. Please' be 
ajjsurea that every attempt will be made to minimize any inconvenience for your 
ins Litu Lion and its departments and to maximize the return to you of helpful and 
interesting^ rc-tr.uilv. . All data received will be held in the strictest confidence 
and no institutional or student identification will appear in any reports. 

We ope that the criterion studies and the statistical tecimiques will be 
:;uccessiul. The need for better rest^arch information in graduate adiT'issions and 
graduate performance evaluation is urgent. 



' incerely , 



Stephen H. Spurr 
Cnairman 



Enclosures 



\ 
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r. ; ^: ' I'' 'ii:- hope t^iiat Liifje JeP'^.i:^ imentG will aupfdy necessary in- 

jiTTiaT, L iih;. r * i c : [;<'.t t,^.' in this stuay . . 

■-; .-ii [ r-'-: t''./ y -ur Toivaruing the • enclosed laateria.l^ to ihe chairman 

■'f tht- «ifj;.-..i-t;ruri:i oT '::c'jn-;mic.'3 at your institution. Please be assured that every 
at.'^-oi'ijit will c>> nade t'j mLnJjn'Lze ari^^ i nconvenience . for your institution and its 
depai't.nent-r: an-.i"'"^ -V n.a.: i.iTrii:e t.he rt^tuj^n to you of helpful ana interesting results, 
ATI o.i^a ro'j'-iv^M.i w ' \ ' >>: ii^i^'i in ihr :j'.rLctest coiU'idencc and no insti vut iona»i 
\)V :'//jdt-?ri t. i J' -f J i. :* :. o'l !. i. wIll :i\j\A.''±r in'any reporis* 

. '.Vo -ha-, ti^ (-riTorion 5tuoi.fii3 and' the L-tatistlcaL techniques will- be 

•:u-^o*';::- fu . , ■Tn*-' no-i.o :' >r better I'osearch inronnation in gi'aduate adjriissions ana 
^rra iua * . J J -rr )!7Tianof: nya Liiation- Is urgent. 

Sincerely,' 



Stephen H. Spurr 
Chairman 
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w i l.j bu e:xocutXHi, onu i n ai^puri.ineiilr* (/■ pisychology and the cjlficr in cijjfwirLrrienl,^; 
of economLcG. It j.:^ our hope that tiiGse departments will 5upp4.y: neces^^ary in- '■ 
roxTTiation and participate in this stutjy. 

We would appreciate your forwarding the enclosed materials to the chairman 
of the department of psychology at your institution. Please be assured that every 
attempt will be made to minimize any inconvenience for your institution and its 
departments; and to maximize the return to you of helpful and interesting resailts. 
^11 data received will be held in the strictest confidence and no institutional 
or student identification wil 1 appear in any reports. 

We hope that" the criterion studies arxd the statistical techniques will be 
successful. Tb ^ need for better reseai^ch information In graduate admissions and 
^^rv'Kiuate perf oriruance evaluation is urgent. 

Sincerely, 

Stephen H. Spurr 
Chairman 

vSHo/hv; 
Krv: \ orjurfy^i 



I 

I 

! 
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! M < I N ( h: r O N, N.J 



0 M r> ■ 1 ( ) 



; . r . • . ../ (■:)'} 
Hit I ^ i>i ( 1 1 \ t \ \ { 



March 




r : 



."rj»-j nember^^ 01 t.he G'^urma^^e Recoi'd Examinations B^ard, whicli ^rormuiates 
p'-'lLclf/j ^j;Tii.din^^ ^hf Graduate Ku'j .;i'd ExaiTiina t ions (GKE) , have been aware that 
res'^^arch inr-Minali m; Gu;)porting tdie use and inteipretation of data on v:hich 
adnlrjriioris aecisi/;n:j art. buoOd is indeed sparse. Suitable critei'ia for evalua- 
ting the outc:oi.'ie5 of graduate education neea to-'be developed, and the relations 
between t hese ci'itei'ia and inl'oima tion available for adnissions aeo Lsion3 must 
be dii5C'jvered. The Board is f;omnitL.ed uo mount systematic ■ criterion develop- 
nf?ri^ 'ir;o acimi:-.;s Lor.s reseai'c[i olToi'ts. 

i:: ea:'Ly N;ve:nbei' 1^7'^-, we contacted the Graduate School youi' Uni'/e.rsity 
j'/xi't.y in hopes that certain recoras might be made available tor research pui'poses 
Hat ei^i.'i? : including the enclosed letter and questionnaire accompan ; the letter 
■:-t initial conttct anu we hoped that ini'ormation describing your department might 
be entered on the quest i'-mna ire and returned to us. We stil l hope so and urge you 
* ; rujf.pty u,-".^ with r.he Lri t' >naat. ion ooLiclted 'm the enclr^sed questionnaire, and to 
pa .'-^ ■■ • i.f 'at.e In the ::tuti,y by making certain oi your^ recoi'ds available to us. 

l he Iriiomation jB the questionnaire will be valuable to us even thoagn it 
;'My n '. T'/^T/e :V?'isibl- i^r you in p'ai't.lcipate » However, we would appi'eciate the 
•pp r*i.ni\v « -J ry to v/e:o:one the aitiicuities that may at thi.^' ti'ae seem to 
jir^-nlh l' y ui' j.^artic Ipat _n. Of couiooe, the ijifomation fi-ori your depai'tment will 
bf- hoL'i in strictest cor^lidence. Your department will not be identified in any 
way in ariy report of tFii.t stud,/ without, your permission. 

In my letter :d' ri)vember I emphasized that the ti'ansfer of da .-a would be 
ai'ran^ed so thatoit ts at a minim.um of difficulty for you. I fully intended ana 
.:^tiil intend that to be the case. 



Sincerely, 




R. F. Boldt 



Chairman^ Measurement Tystems 
Researcli Group 




EKLC 
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November 6, 19Y0 



The members of xAie Graduate Record Examinations Boai'd, which formulates 
i>o I ij Le-i guiaing Gi'aduate Record Examinations, have for some time been aware 

That research findings: on the validity of our examinations are inadequate, being 
lew and not reprenunr.ative . We would like to see that more studies are accomplished, 
t)UT. Ln attempting t,') no so encounter criterion problems and prob],ems arising from 
the :5carcicy of sui^-^bLe data. Her, b we are developing plans for criterion research 
and also in i tia t iri^3 methodological studies on techniqires for pooling data for use in 
■ iirferent 'rece Lvinp, I ns t. itut i ons . This letter requests your participation in a 
validation study be i n^^' c onduc t od by the Educational Testin^i Service for the Graduate 
r<-.*co: a Examinat,ioni. f^oar-d. The study will compare a" variety of complex regression 
:;y.)tnms, '"'i<)\,h Bayr:;iin an^i least squares, and will be conducted in departments of 
♦^ •conofM ic s . 

iho rioth)(i^; u;-*-.: wiii, hopefully, prove to tolerate situations where veiy few 

' : 'J :> ■ I I i I ar- 1 ' ^ ^ m' srh' ^o I . 

Vir: \^; AiI:[ i lk'- ' u:>.' uat,a relLeo^Lng yoUx ex]^^.^:' \';nce omi are enclosing a form 
>-r:L'-:r. V'ju ':ari LritJLr'iw; the type and avail ab i ' i'.y of uata you may h. '^e available-. 
r;'iV' til in* [n '.:;in^; youi' cia ta e'en if there ive not many cases. Based 
\ your r^'.-j iii^^a: w \-:'. 11 f r'^j; .>se a means ol urans leiu lug data fr-cfu youi" files to 
•urs with a lainirjun >:' mmp i Loar. ion I'or you. 

The neeu t' u' o"*'- r research infomatiori .n graduate adjiissions is urgent and 
\::i.s s^ucl>* may rie i ] u, rrovJde It. We hope you will participate in this study. 



Si. erely, 




R, F. Boidt 

ChaiiTTianc Measurement Syst.ei;.-. 
Re search Gi' oup 



surt 
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IM<IN1 I ION, N.J OHn-K) 
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March 1, 1971 




Dear Sir: 



The members of the Graduate Record Examinations Board, which formulates 
policies guiding the Graduate Record Exajtiiiiat ions, have for some -time been aware 
that research findings on the validity of our examinations are inadequate, being 
few and not epresentative . We would like to see that more studies are accomplished, 
but in attempting to do so encounter criterion problems and problems arising from 
the scarcity of suitable data. Hence we are developing plans for criterion research 
and also initiating methodological studies on techniques for pooling data for' use in 
different receiving institutions. This letter requests your participation in a cross- 
validation study being conducted by the Educational Testing Service for the Graduate 
Record Examinations Board. The study will ccjnpare a vai^iety of. complex regression 
systems, both Bayesian and least squares, and will be conducted in departments of 
psychology. 

Thr methods used wj.il, hopefully, prove tolerate situations where very few 
cases are available per school. 

We would like to use data reflecting your experience a^e enclosing a form 

on which you ca-p indicate the type and availability of data /ou may have available. 

We do -have an interest in using your data even if there a ■ ot many cases. 'Based 

on your resj^^onses we will propose a means of transferring ^ a from your files to 
ours w.LMi a rrinimum of complication ;for you. 

The need for better' research information in graduate admisr-ions is urgent and 
thiij ntucjy may help us pi ovide it. We hope you will participate in this study. 



Sincerely, 




R. F. Boldt 



Chairman, Measurement Systems 
Research Group 



hncioipu^e 



EKLC 



